For a while I have been thinking about leaving Mixcloud. Its nothing personal, I think its a great service for djs but its clear their business model is starting to interfere with the listening experience.
I had a thought about what Mixcloud do for me and decided these are the key things.
- Hosting the full mix with limited rights problems (unless you do something like play a track from the same author twice)
- The community of people and djs in one place
- Ability to see the actual tracks within the mix
With this in mind, I thought I’d see what else was out there as I’m lucky enough to be sitting on a large enough internet connection to host my own mixes and become a node on a larger decentralised network. If it was build like the fediverse, that could solve the community side too? After looking around for alternatives I found a new upstart called funkwhale. You may have seen I mention it recently in a previous blog.
Maybe 1 and 2 could be solved but what about 3?
The last part of the puzzle seemed to be the track problem, as you want seamless playback but get an idea of what you are listening to. Mixcloud does this via metadata, which you can create via a slider over the audio waveform. Some DJ systems create this for you like the Pacemaker for example but that metadata is lost in translation I found. If only there was a standard way to define areas of a mix without slicing the audio mix up?
For example, here is a ambient mix I liked recently.
Its by Tonepoet and even they have gone through the effort of adding this metadata to their personal site complete with timings. I do a similar thing but without the timings (which I really should have added since I had them all and entered them into Mixcloud manually.
I looked a number of things including a bunch of playlist formats including pls, m3u and xspf. Even looked at smil and asx to see if they would help, but their problem was player support. The issue seemed to be they all treated their smallest objects as physical files rather than subsets of files. I did buy into xspf thought it was close with this extension.
The extension element allows non-XSPF XML to be included in XSPF documents. The purpose is to allow nested XML, which the meta and link elements do not. xspf:playlist elements MAY contain zero or more extension elements.
<playlist version="1" xmlns="http://xspf.org/ns/0/" xmlns:cl="http://example.com"> <extension application="http://example.com"> <cl:clip start="25000" end="34500"/> </extension> <trackList /> </playlist>
Close but not quite right and player support for extensions was going to be low. This is when I rethought the problem with something like .nfo files and found .cue files. Here is an example…
REM GENRE Electronica REM DATE 1998 PERFORMER "Faithless" TITLE "Live in Berlin" FILE "Faithless - Live in Berlin.mp3" MP3 TRACK 01 AUDIO TITLE "Reverence" PERFORMER "Faithless" INDEX 01 00:00:00 TRACK 02 AUDIO TITLE "She's My Baby" PERFORMER "Faithless" INDEX 01 06:42:00 TRACK 03 AUDIO TITLE "Take the Long Way Home" PERFORMER "Faithless" INDEX 01 10:54:00 TRACK 04 AUDIO TITLE "Insomnia" PERFORMER "Faithless" INDEX 01 17:04:00 TRACK 05 AUDIO TITLE "Bring the Family Back" PERFORMER "Faithless" INDEX 01 25:44:00 TRACK 06 AUDIO TITLE "Salva Mea" PERFORMER "Faithless" INDEX 01 30:50:00 TRACK 07 AUDIO TITLE "Dirty Old Man" PERFORMER "Faithless" INDEX 01 38:24:00 TRACK 08 AUDIO TITLE "God Is a DJ" PERFORMER "Faithless" INDEX 01 42:35:00
Perfect, so I took one of mixes, Quiver in the underground and turned the .nfo file into a .cue file.
REM GENRE Tech Trance REM DATE 2019 PERFORMER "Digital Italic" TITLE "Quiver in the underground mix" FILE "Quiver in the underground mix.mp3" MP3 TRACK 01 AUDIO TITLE "Activator, I know you can (That kid chris mix)" PERFORMER "Whatever girl" INDEX 01 00:00:00 TRACK 02 AUDIO TITLE "Air traffic (Erik De Koning remix)" PERFORMER "Three drives" INDEX 01 02:07:00 TRACK 03 AUDIO TITLE "Chinook" PERFORMER "Markus Schulz pres. Dakota" INDEX 01 07:25:00 TRACK 04 AUDIO TITLE "Opium (Quivver remix)" PERFORMER "Jerome Isma-Ae & Alastor" INDEX 01 10:32:00 TRACK 05 AUDIO TITLE "Surveillance" PERFORMER "Jordon Suckley & Kutski" INDEX 01 15:24:00 TRACK 06 AUDIO TITLE "Nitric (Division one remix)" PERFORMER "Hybrid system" INDEX 01 19:02:00 TRACK 07 AUDIO TITLE "Circa-Forever (Galen Behr & Organ Nilsen remix)" PERFORMER "Rapid eye" INDEX 01 23:55:00 TRACK 08 AUDIO TITLE "Opulence" PERFORMER "Simon Patterson" INDEX 01 30:05:00 TRACK 09 AUDIO TITLE "J'ai envie de toi (Protoculture remix)" PERFORMER "Armin Van Buuren presents Gaia" INDEX 01 35:10:00 TRACK 10 AUDIO TITLE "Z.I.T.A (M.I.K.E's progressiva mix)" PERFORMER "Hiver & Hammer with Funabashi" INDEX 01 39:30:00 TRACK 11 AUDIO TITLE "Kubrick (Extended mix)" PERFORMER "Jerome Isma-Ae & Alastor" INDEX 01 42:01:00
Tried it out on a number of media player and they all worked except Plex.
There is a very good chance if I was to run my own funkwhale node/server I could set it to index .cue files and playback the mix in a seamless way like Mixcloud? Theres only one way to find out really… Get Funkwhale installed!
On a side note I am slightly kicking myself now because I entered all that metadata into mixcloud but never stored it myself. I’m going to need to go through 90 of my own mixes and convert my .nfo files into .cue files.
I have the SQLite Database for my pacemaker device with the actual real timings but I need to identify which mix is which one (another thing I should have done, as changed the names of the mix depending on many things). I also hoped mixcloud’s developers guide would come to the rescue but it looks like maybe a GDPR request is my only option if I want the metadata for my mixes?
I mentioned a while ago how I was slowly migrating away from Mixcloud as their business model is starting to impinge on people listening to my mixes and I’m not so keen on that. I already mentioned trying to get Funkwhale working and using cue files.
While looking through my mixes I noticed I really didn’t do a great job with some of the metadata. While I had spent most of my time adding to the metadata to Mixcloud (not ideal)
In Mixcloud (once logged in) there are some human friendly urls which I was able to grab images from. The key one being the upload edit page – https://www.mixcloud.com/upload/{username}/{mixname}/edit/ for example https://www.mixcloud.com/upload/cubicgarden/follow-me-into-the-fading-moonlight/edit/
My plan was to manually copy the times into my newly written cue files but while talking to Jon about it, he said give him 5mins and he could knock up a script to pull the values out of the HTML page. I thought about it before but using XSLT, however noticed there is a lot javascript rendering making things difficult.
Jon’s quick script written was just what I needed.
#!/usr/bin/env python3
import csv
import sys
from collections import namedtuple
from typing import List
import bs4
from bs4 import Tag
SongInfo = namedtuple(‘SongInfo’, [‘number’, ‘artist’, ‘title’, ‘time’])
def load_html(filename: str):
with open(filename, ‘r’, encoding=’utf-8′) as fo:
return ”.join(fo.readlines())
def extract_song_info(song: Tag):
try:
number = song.find(class_=’section-number’).text
artist = song.find(class_=’section-artist’).text
title = song.find(class_=’section-title’).text
time = song.find(class_=’section-time’)[‘value’]
result = SongInfo(number, artist, title, time)
print(f’Extracted {result}’)
return result
except AttributeError:
print(f’Error with item {song}’)
return None
def parse_table(input_html: str):
soup = bs4.BeautifulSoup(input_html, features=”html5lib”)
songs = [row for row in soup.find_all(class_=”section-row”)]
return [x for x in [extract_song_info(song) for song in songs] if x is not None]
def save_to_csv(file_name: str, songs: List[SongInfo]):
with open(file_name, ‘w’, encoding=’utf-8′) as fo:
writer = csv.writer(fo)
for song in songs:
writer.writerow(song)
if __name__==”__main__”:
if len(sys.argv) != 3:
print(‘Usage: extractor.py [input_html_file] [output_csv_file]’)
html = load_html(sys.argv[1])
songs = parse_table(html)
save_to_csv(sys.argv[2], songs)
print(f’Saved to {sys.argv[2]} successfully – Done!’)
With it and the HTML pages, which I almost got with Chromedriver, again thanks to Jon, but I couldn’t be bothered to sort out the cookies, etc. I quickly wrote a quick /dirty bash script and fired up a terminal.
#!/bin/bash
./extractor.py $1.html $1.csv
# Verify
echo Details for $1
I thought about modifying Jon’s script to generate the cue files directly bypassing the csv file but decided I should just get them all done. Because I still need to get funkwhale going.
I did notice the edit page doesn’t include genre or the year of the mix, but I can live with that, for now… Scraping web pages is certainly a throw back but its better solution that what I originally was thinking.
This will teach me to sort out my own house of data!