Rationalisation of Woof-CE

It has been over 2 and a half years since Barry handed over Puppy to the community. In that time we have made many improvements to woof to modernise Puppy and hopefully provide a more modern interface.

However, there is still much work to be done.

While on the surface there doesn't seem to be much happening, behind the scenes development is actually heating up! We have a new member on the team who has been very busy adding and editing thousands of lines of code along with the old faces of zigbert, iguleder and gyro (GitHub handle gyrog) updating and refining code and 666philb (GitHub handle mrfricks) churning out exciting new xenialpup beta releases. I myself am working on a slacko-631 bugfix release (but I may bump the version number to reflect the massive changes in woof).

Without further ado I'd like to welcome the said new member of the woof-CE team wdlkmpx (GitHub handle - a.k.a. jlist on the Puppy forum). He has been pushing several pull requests lately and they have all been sound so I decided to make him a full member. So if something goes wrong some of the heat is off me!

Puppy development isn't just woof-CE. An important part of Puppy is the initrd.gz ('initial ramdisk', for want of a better term) that takes care of the booting of Puppy after the bootloader and kernel have done their respective jobs. I started the initrd_progs woof repo to reliably reproduce the initrd programs which wdlkmpx has improved substantially. << Read totally re-written!

So where to now?

Unfortunately, we haven't many builders to test the various distro compatibility. While we all would love Puppy to be our full time jobs, it doesn't pay the bills! We are going to rationalise the number of distros that are supported by woof-CE. Don't panic though. We are going to preserve all the old code in a Legacy branch so that if an enthusiast does come along wishing to revive one of the relegated distros then they don't have to start from scratch.

Distros facing the chop are
• arch
• mageia
• pet-based
• T2
• scientific

as well as old unsupported upstream projects such as Lucid Lynx and older Slackware and Debian releases. Old puppy 2, 3 and 4 repo databases will be removed from woof (not the main repo).

Rationalising of the build scripts is also a priority. A functions library would make life a lot easier and the code more readable. Numerous other improvements need to be made.

Consideration to shifting firmware out of the zdrv and making fdrv (system sfs files) load by default is under review. This will also require a new kernel structure, although old kernels (from the huge style) will still work. Old kernel 'pets' will be removed.

Also, the integration of the initrd_progs into woof-CE would be desirable. The resulting initrd.gz would be smaller and faster with the latest busybox and supporting programs included.

I would also like to see some C sources in woof for core programs so that they can be compiled on the fly in a chroot. This should not be a problem when native building but may have to be addressed for a cross build.

Applications should be grouped to make it easier for builders to make a minimal or an "everything including the kitchen sink" Puppy! The woof GUI will need substantial improvements.

These changes will take time and effort. They are designed for maximum compatibility when a distro is built and hopefully reduce compatibility issues.

I look forward to any comments and may even start a forum thread about this for those who don't want to register here.

Posted on 4 Jun 2016, 18:12 by 01micko - Categories: Woof-CE
2 Comments - Edit - Delete

pMusic DB and file format

pMusic is an audio-player using a database (DB) to keep track of meta-information of your music. This is info like:
  • ID3-tags (artist, title, album, ...) that are embedded into the file.
  • links to lyrics, albumart, ...
  • Stream info like format, samplerate, bpm, ...
  • User listening activity.

  • While most players depends on an external DB like Mysql or Mariadb, pMusic has its own, very simple, internal DB. Basically it is a textfile with 21 columns separated by a |, where each line contains info of one file. When reading from the DB (ie. when searching), we just grab the line(s) corresponding to our request.

    To make the infra-structure simple; the DB, the sourcelist (search-result-field) and the internal file format (*.pmd), uses the exact same formating. That means, we can dump the raw output of the DB to the gui, a favorite list, or ie. to the smart-add function. This helps a lot when it comes to speed. Sometimes you might forget that pMusic is not compiled binary, but just a bash script.

    GeekNote: To be technically correct, the sourcelist and *.pmd file contains only the first 15 columns of the DB format. They have no use of the last items of info.

    The DB structure

    Let's look at one line of my DB:
    /mnt/sdb1/musikk/Tindrum - Drums Of War.mp3|Tindrum|Drums Of War|Drums of war|1|1988|Hard Rock|Live||mp3|175|04:02|117|/mnt/sdb1/musikk/Tindrum - Drums Of War.mp3|gtk-audio|2aa59ada-3dd8-40b7-878a-bb682fca7b30|479933cd-d787-4126-8315-7749ca4781a1|Tindrum|/mnt/home/pmusic storage/albumart/Tindrum - Drums Of War.jpg||,1357415329,1357417187

    This line holds all info we got about this track. Instead of explaining what it all means, just open up the DB-editor found in Menu > Music sources > My Music. Here we can look at, and edit all info in the DB.

    Most of this is straightforward, but 3 things might not seem logical:

  • The Command (column 1) and the Path (column 14) is equal. The Command is executed when adding stuff to the playqueue. The Path is what it says - the local path. Because the example shows a local file taken from the DB textfile, the command is equal to the path. This is not the fact for an url pointing to a radio-station web address.

  • The last column, Timestamps, is not self explanatory. Each time you has played a song, the DB adds the given time in seconds since 1970 to the end of the line. This is used to provide better sorting depending on your preferred music. And, to search for your most played songs ie. in February.

  • Column 7 - Rate - is empty, and it is always empty in the DB textfile, but not in the sourcelist (left pane). The rating is equal to number of timestamps, and is calculated in func_rating. This calculation also change the icon depending on the song's rating. Geeknote: This is a great example of the blinding speed of awk.

  • Writing to the DB

    pMusic allows 2 major ways to write info to the DB. Either by (1) user interaction or the default (2) silent mode.

    (1) user interaction
    The static way is to start the indexer (Menu > Music sources > My Music) which scans your system for audio-files and adds the info to the DB. Depending on your music collection, this might take a long time, but place all available information (ID3-tags and stream info) to the DB at once.

    (2) silent mode
    The default dynamic behavior is to store info as you play. When you press the Play-button, pMusic
  • checks the file for ID3-tags and stream info.
  • searches the web for lyrics, albumart and artist info. (Must be activated in the preferences.)
  • tries to build an album-list if tracks are available on your system. (Must be activated in the preferences.)

  • If using mode (1), we still miss info from the web and album lists, so mode (2) will kick in anyway for missing info.

    All this info must be written to the DB - but that's not straightforward - we can't just start writing...
  • To avoid parallel writings to the DB, there exist a queue-system (DB is locked while writing).
  • To avoid a long queue (many locks are resource hungry), the info is passed to a DB-stack. The DB-stack can hold several info-capsules.
  • Writing to the DB, empties the stack in one operation. This avoids the queue to grow.

  • The stack is found at $HOME/.pmusic/tmp/mymusic_stack.
    The merging is done at func_index -merge_stack.

    Keep the DB in shape

    In addition, there are mechanisms to keep the DB in shape. When you add a file to the playqueue, pMusic checks it in func_add - functions check_source and fix_db.

    If the added file is not already in the DB, ask user to scan the source directory for new music. This message shows up embedded in the main window to not disturb the user too much. It could be that we don't want to respond to this message, and that is just ok.

    If the added file has its line in the DB, but not present in your system, something should probably be updated, and this message below will show up. But, if the filename is found in the DB with another path than the defined, pMusic will play the alternative source instead. Therefor, this box is not often seen in recent versions of pMusic.

    The Playlist file format

    I mentioned earlier the internal file format *.pmd used by ie. the favorites structure. But, for the user, the pMusic format is the *.pmu found in the save box. You can choose to save to an external format as *.m3u or *.pls, but to get the full power of pMusic, *.pmu is the recommended choice. This because it cooperates well with the DB.

    *.pmu contains the same info as in the playqueue. This makes it possible to dump a *.pmu into the playqueue, while a *.m3u has to be converted. Since all information in a *.pmu is grabbed from the DB, we can quickly revert, and grab the full track-info back from the DB. This is seen when expanding a *.pmu in the sourcelist (left pane). - We see full track-info including rating calculation. When expanding a m3u/pls we only see the actual info inside the file.

    When looking at the playqueue in the gui, you see 3 columns:
    Length | Track title | Path

    In fact there are 2 more hidden columns, and all are part of the *.pmu file:
    Icon | Length | Track title | Path | ID

    The Icon holds gtk-play when the play-icon is shown, else it's empty. This will of course always be empty inside a *.pmu file
    The ID holds an unique id-number together with the path. This is necessary to avoid conflict when playqueue contains several instances of the same song. The ID keeps track of which of the similar songs you are playing/moving/editing.

    Unlike *.m3u, the *.pmu playlist does not handle relative paths. This to stay close to the DB format. Instead pMusic will search for an alternative path if playing track is not found. The statusbar will show this information during search. Code found in func_progressbar.

    The short story of the pMusic file formats is:
    *.pmd mirrors the DB
    *.pmu mirrors the playqueue

    This is the basic structure of the pMusic data format.

    Posted on 9 Jul 2016, 18:54 by zigbert - Categories: Development
    No comments - Edit - Delete

    Upload to Woof via web interface

    This describes a very specific challenge.
    How to upload a package of files to Woof-CE at github via the web interface.

    The web interface is suited for doing simple things, and has many limitations over the CLI interface. An advanced user of git would probably laugh at us using the simple variant. But when rarely using git, and most of the time just editing existing files, it is just perfect to do the stuff in the web-browser.

    The challenge is when we want to upload packages containing lots of files and directories. How do we do this...

    1. Create new files and directories

    - Go to the source directory - in this example /woof-code/rootfs-packages/.
    - Press the Create new file button.

    - Write the name of first directory in the textbox - here frisbee.
    - Enter / to create the path. The textbox moves right, and are ready for new input.
    - When reached target directory, add filename in textbox, and add file content in the editor.
    - Add pull request by pressing the Propose file change / Commit changes button.

    2. Upload files to existing directory

    - Go to the source directory - here /woof-code/rootfs-packages/frisbee/etc/.
    - Press the Upload files button.

    - Add files by dragging them from your file-browser.
    - You'll see your files get added at bottom (marked red).
    - Add pull request by pressing the Propose file change / Commit changes button.

    Posted on 27 May 2016, 01:35 by zigbert - Categories: Woof-CE
    1 Comment - Edit - Delete

    LxPupSc - new version may-17

    LxPupSc-17.05.1T with kernel 4.10.13 is available.

    See this blog entry for more description of LxPupSc

    Posted on 2 May 2017, 02:29 by peebee - Categories: Puppy Woof-CE Release
    No comments - Edit - Delete

    The gtkdialog lib

    Did you know about the gtkdialog lib?

    Puppies built from Woof-CE uses /usr/lib/gtkdialog/ to unify the look of builtin GUIs. Together with the new vector icon set (in /usr/share/pixmaps/puppy), the gtkdialog lib made it possible to give Puppy a facelift without adding any bloat to the core.

    The lib is simple and small. It is meant to reduce the code for repeating requests (as message-boxes), and unify the look for graphical widgets (as scalegrips and svgs). A complete guide how to use the different snippets are found inside each file (open in text-editor). It contains 3 different categories:

    1. BOX
    These return running gtkdialog boxes with requested info. Gtkdialog is not complex at all, but dead simple boxes often used by Puppy, can be set up with a oneliner. This is not meant as a replacement for Xdialog.

    There are atm 4 different types of boxes:

    /usr/lib/gtkdialog/box_ok Title error "This is an error"

    2. XML
    While the box-code produces a running gui, the xml-code is gtkdialog code meant to be embedded inside the parent gui. These are used to unify the look, and is cooperating with pTheme to follow the global theming.

    There are 4 xml-snippets available:

    Example: - Note the 'xml_info gtk' that activates the gtkdialog-theme set by pTheme.
    export GUI='
    <vbox space-expand="true" space-fill="true">
    '"`/usr/lib/gtkdialog/xml_info fixed puppy_config.svg 60 "This is the header used in Puppy these days. It handles <b>markups</b>."`"'
    <button ok></button>
    . /usr/lib/gtkdialog/xml_info gtk #build bg_pixmap for gtk-theme
    gtkdialog -p GUI

    3. SVG
    The svg category returns svg-code to embed in the gtkdialog code. Today, there are 3, but the knob shown in previous post should probably be added here. As expected, the svg bars and text follows the global theming set by pTheme.

    echo '
    <timer visible="false">
    <action>/usr/lib/gtkdialog/svg_analogclock 300 > /tmp/clock.svg</action>
    <pixmap space-expand="false" space-fill="false">
    <input file>/tmp/clock.svg</input>
    </vbox>' | gtkdialog -s

    Posted on 19 May 2016, 20:19 by zigbert - Categories: Development
    No comments - Edit - Delete

    gtkdialog knob

    Some time ago mister_electronico on the Puppy forum started the discussion of using knobs in gtkdialog code. I gave some feedback but the code never entered a final state. I have now revisited the code, and plan to use it myself.

    So, here is a simple test-script to show how it works. All in all, it's just a svg that rotates the dot to show the changing value. Atm, the knob rotates when dragging mouse up/down on the knob. Maybe this should be extended to left/right actions as well. Please check out the code; it should be easy to understand with some basic bash/gtkdialog/svg knowledge.

    The code:
    export KNOB_WIDTH=120
    export KNOB_COLOR="#A5E8B1"
    export CARD_NR=0 #'aplay -l' shows your cards
    export SPEAKER=Master

    #initial audio level
    TMP="`amixer -c $CARD_NR get "$SPEAKER" | grep -m1 'Left:' | cut -d '%' -f 1 | cut -d '[' -f 2`" #volume level
    ANGLE=$(($TMP*250/100)) #convert from % to angle (0-250)
    echo $ANGLE > /tmp/knob_angle

    draw_knob (){ #$1=angle/value, $2=width, $3=color, $4=dot-color
    echo '
    <svg width="'${2}'" height="'${2}'">
    id="RG1" cx="'$((${2}*30/100))'" cy="'$((${2}*30/100))'" fx="'$((${2}*30/100))'" fy="'$((${2}*30/100))'" r="'$((${2}*45/100))'"
    <stop style="stop-color:#393939;stop-opacity:1;" offset="1"/>
    <stop style="stop-color:#888;stop-opacity:1;" offset="0"/>
    <circle style="fill:#222;" cx="'$(((${2}/2)+(${2}*4/100)))'" cy="'$(((${2}/2)+(${2}*4/100)))'" r="'$((${2}*46/100))'"/>
    <circle style="fill:#777;stroke:#444" cx="'$((${2}/2))'" cy="'$((${2}/2))'" r="'$((${2}*46/100))'"/>
    <circle style="fill:url(#RG1);stroke:'${3}';stroke-width:'$((${2}*2/100))'" cx="'$((${2}/2))'" cy="'$((${2}/2))'" r="'$((${2}*39/100))'"/>
    <circle style="fill:'${4}';" cx="'$((${2}*28/100))'" cy="'$((${2}*74/100))'" r="'$((${2}*4/100))'" transform="rotate('${1}' '$((${2}/2))' '$((${2}/2))')"/>

    update (){ #get current, relative position of cursor
    read ANGLE < /tmp/knob_angle
    Y="`getcurpos | awk '{print $2}'`"
    read Y2 < /tmp/cursor_y
    ANGLE=$(($ANGLE+(($Y2-$Y)*2))) #the knob's value range is equal to moving mouse up/down 125 px.
    [ $ANGLE -lt 1 ] && ANGLE=0; [ $ANGLE -gt 250 ] && ANGLE=250 #keep value inside range
    echo $Y > /tmp/cursor_y
    echo $ANGLE > /tmp/knob_angle
    draw_knob $ANGLE $KNOB_WIDTH "$KNOB_COLOR" "#eee" > /tmp/knob.svg
    amixer -c $CARD_NR set "$SPEAKER" $(($ANGLE*100/250))%

    export -f draw_knob update
    draw_knob $ANGLE $KNOB_WIDTH "$KNOB_COLOR" "#eee" > /tmp/knob.svg

    echo '
    <vbox margin="20">
    <timer visible="false" milliseconds="true" interval="100" sensitive="false">
    <input file>/tmp/knob.svg</input>
    <action signal="button-press-event">getcurpos | awk '"'"'{print $2}'"'"' > /tmp/cursor_y</action>
    <action signal="button-press-event">enable:TIMER</action>
    <action signal="button-release-event">disable:TIMER</action>
    </vbox>' | gtkdialog -s

    Be aware that this knob depends on the getcurpos command. This is included in all recent Puppies. But I doubt that you'll find getcurpos in most other distros, so I see this as Puppy-code. You should probably avoid this code if you plan to offer your gui outside the kennel.

    This info is added to the Tips and tricks post

    Posted on 10 May 2016, 04:09 by zigbert - Categories: Development
    No comments - Edit - Delete

    Pages: ... [3] [4] [5] [6] [7] [8]