tirsdag den 17. januar 2017

A note on Home-Assistant and SD-cards

(Beware of) Killing your SD-card

If you use hass on a raspberry pi, make sure you don't write a lot to your SD-card.

Well this warning goes for anything running of a sd-card, but what I've noticed however is that hass really writes a lot to the card.

My fist SD-card didn't last for more than two weeks, but it was a cheap one, and I've used it for many other tasks, so I though it had just reached it's end of life.

But when my new 16 GB kingston card died after 2 months, so I though I'd better take a closer look at what was going on.

In the default configuration file, two components are enabled by default:
logbook and history.

recorder

This component is used by history (see next section).
Url: https://home-assistant.io/components/recorder/

It specifies what backend to use, and how mush data to keep.

history

If we take a look at history (https://home-assistant.io/components/history/), it creates a file called "home-assistant-v2.db" in your configuration directory (unless you're specified another path in the recorder component).

It logs state changes, so only writes to the database when something changes.

Note that you can exclude certain entries from being logged.


Logbook

This element writes
It writes to this file EVERY second!
It does this to log the state of your system, but COME ON, every second!?!?!


A better way would proberly be just to log state changes.
It would both reduce the number of entries, and the load on the SD-card.
Now I don't know yet how logbook logs it's entries, but it might do the same thing.

Fixing it

A couple solutions are available (that I'm aware of):
  • Log to non-flash storage
    Either attach a harddrive, or over the network.
  • Disable the components
    Just comment-out the lines in the configuration file.
  • Save the file(s) on a RAM-fs and then once a while save it to disk
    It's easy to use tmpfs.

For now I've just disabled the components.


fredag den 23. december 2016

Using Home-Assistant to smarten my home and playing around with IoT, and even dumber things (433mhz)

Intro:

I admit that I'm somewhat of a geek, and one of my many intersts is home-automation, robotics and IoT, so ofcourse I NEED to have this in our house.
The only problem is that:
  • I don't really want to pay all that much for it
  • Don't want to be locked into one vendor/technology
  • Would like it to do so much more than any commercial solutions I'v found so far can do.
  • I would like to integrate with what we already got/have at home.

So far I been using a raspberry pi with some 433Mhz transmit and receive modules (link).
The TX module works ok, but the receiver is bad, and only works within ~3 meters. So I upgraded the receiver to a RXB6 (link), and now the range is at least 30 meters (even without any antenna soldered onto the PCB).

So what do I control?

  • Power-sockets (link):
    We have a bunch (+20) of 230V power-sockets (3 sockets cost less than 20$ US, including a remote).
    These control some of our lights.
  • Some relays (link):
    These help toggle other lights, and other stuff that either just needs a small flick (as they only turn on for a short time, and back off again), or don't run on 230V.

On the software side, here's how I controlled it

  • rc-switch (link)
    To send and receive codes.
  • IFTTT (link)
    To automate some of the lights, so the outside lights turn on at sunset, and off at sunrise.
  • PHP to implement some webservices, so IFTTT could talk to rc-switch, and also our smartphones could be used as a remote.
  • Simplepush (link) to push out notifications from my system to phones


What about input/sensors?

Oh yes, I've got that too:
  • PIR sensors (link)
    For motion detection.
    Though we have a 6 kg cat, it's never triggerd a false alarm from these, which is quite nice.
  • Door/Window sensors (link):
    These a magnetic sensors, and are placed on doors and windows. 


Now there are some problems with this setup/system

  1. Automation is hardcoded, or not easy to change.
  2. There's no nice interface.
  3. I'm missing a whole lot of features.

I would like a better interface, and a mobile client.
I would also like to control and monitor more of the stuff that I already have.

For instance, I would like to monitor the temperature/humidity in our basement, so we can avoid any potential problems with moisture.

I also have several RGB led strips that are currently controlled by a crappy IR remote, so they aren't really used, even though they are really good at setting a mood.

The smoke detectors we have it this house are linked, so if one ones, they all sound. They emit on 433Mhz as well. I want these integrated as well.
And while we're at it, why not use these as sirens if/when the alarm goes off.

Our doorbell, why not also integrate that (433 wireless as well)?

While I like developing the backend, and device stuff, web and mobile interfaces aren't really my thing, so I've been looking for something to help me in this deparment.

A note on the price

Well, as I said in the beginning, I think the solutions on the market today are too expensive.
Here in Denmark (and most of europe) there an alarm company called Verisure, that also frequently calls me. They offer Smoke detectors, PIR's, door/window sensors, camera's and remote power sockets.
They don't mention prices on their website, but they offered me a package containing (the home plus):

  • 2 camera's (with PIR)
  • 2 smoke detectors (with temperature and humidity logger)
  • 3 door/windows contacts
  • 1 remote power socket
  • 1 voice box
  • 1 keypad
  • 1 keyfob
  • 1 central unit (with battery backup)
And the price you ask?
986 US$ (yes, almost one thousand US dollars - and no, I'm not missing a comma).
And then you of course also need to pay for the installation: 281 US$
And lastly (whch is the "best" part), it's not when you equipment! You're just renting it, and if you cancel your subscription you'll have to deliver it all back.
Now the subscription is "only" 45 US$ pr. month.

Ouch!

Okay, you'll never have to worry about changing the batteries, because that's included in the subscription.

Their website does not mention any prices, but always just offers a "lets call you" option (which is why they keep calling me, as I did this once).

During one of these calls I asked what an extra smoke detector costs, and this is what I got: One-time fee of  140 US$ (999 dkr.), and a monthly free of 5 US$.


They have good rating on trustpilot, seem like serious, and know what they talk about, but for those prices, they really also should!

So lets take a look at the prices, for a home-made alarm (of course all prices are one-time, and the equiptment is yours afterwards, and there's no mondtly fee. You just need to change the batteries once a while):

  • Smoke detectors: Oddly expensive, but I have yet to find these "linked" ones on ebay) - link: 16 US$ - 109 dkr.
  • PIR - link: 4 US$ - 28 dkr.
  • Door/window sensor - link: 3 US$ - 17 dkr.
  • Temperature/Humidity sensor - link: 6 US$ - 37 dkr.
  • RFID/NFC tags - link:  4 US$ - 27 dkr (for 10!) or less than 0,5$ each.
  • Power socket: - link: 14 US$ - 94 dkr. (for 3, including remote!) or 4$ each.
And you the controller you'll need:
  • Raspberry Pi (I'm using a RPI2) - 35 US$
  • Micro-sd card (or plain if you use a RPI1) - 5 US$
  • Power adapter - 5 US$
  • 433 RX module (RXB6) - 2 US$
  • 433 TX module (link) - 1 US$ (for a TX/RX pair)
So a comparable system would be:
  • Smoke detectors: 2x16 = 32 US$
  • PIRS = 2x4 = 8 US$
  • Door/windo sensors: 3x3 = 9 US$
  • Temperature/Humidity sensors: 2x6 = 12 US$
  • Power sockets: 1x14 US$
  • Controller:
    • Raspberry pi3 - 35 US$
    • SD-card - 5 US$
    • Power adapter - 5 US$
    • 433Mhz TX/RX modules - 3 US$

In total: 123 US$ (if you need all of the things)
SD-card, power adapter, and if you're reading this, raspberry pi, you might already have.

Now of course they are not 100% comparable.
For instance I'm using a old google nexus 10 tablet (with nfc) as control panel.
Otherwise you can use your phone as control panel.

One important thing the our home-made system is missing are the camera's, but I'll come back later and show you how to integrate these (any ip-camera will do, or you can even use a old usb webcam, and connect it to the rpi).


Again the whole point is to use sensors that are cheap and what you already got at home.


Meet Home-Assistant:

As I don't like playing around with web-interfaces as much as other software I've been looking for open-source solutions to help me out.
I've found two interesting projects:
openHab and Home-Assistant.
openHab is implemented in Java, and HA in python.

I haven't used python before, but HA seems to have more momentum, so I choose this.
Out-of-the-box it does not have the capability to receive 433 codes from rc-switch, you I'll show how to implement a component for this.

Here's my notes on setting it up.


HA Setup-Guide:

Requirements:

  • 433Mhz modules (RX6B receiver is best)
  • Raspberry pi
  • switches/pir's/more

Software install:

Install arch linux

Go to https://archlinuxarm.org/ and follow the instructions to install arch linux on your pi flavor (I'm using a RPI2).

After arch linux for ARM is installed on the SD-card, boot the rpi, and ssh into the booted device (or login locally) as alarm:

Change default alarm
passwd
And root password
su
passwd

Update pacman 
pacman -Syu

Localize system

timedatectl set-timezone Europe/Copenhagen
hostnamectl set-hostname rfcac
nano /etc/locale.gen
(uncomment da_DK.UTF-8)
locale-gen
localectl set-keymap dk

Install Home-Assistant

Install python 3:
pacman -S python python-pip

If you what to use netdisco (which is also a part of the first demo), you'll need gcc as well:
pacman -S gcc 

Install Home-Assistant (HASS):
pip3 install homeassistant 

You can check that it works by running:
hass --open-ui

Note that that it takes a while for hass to first start as it needs to download missing modules. This is also true when adding new modules to the configuration.

Test the install:
Point your browser to http://localhost:8123


We'll create a systemd unit file to run hass on boot (/etc/systemd/system/hass.service):
[Unit]
Description=Home Assistant
After=network.target
[Service]
Type=simple
ExecStart=/usr/bin/hass
[Install]
WantedBy=multi-user.target

After creating the file, re-read all unit files:
systemctl daemon-reload
And make sure it's started when your raspberry boots:
systemctl enable hass


Components
In Home-Assistant we'll be using two components to control our 433Mhz traffic.
One is built-in, and is called "Raspberry Pi RF Switch".


rfcac
The other is a homemade one, if called rfcac.
You'll need to create the file rfcac.py in a custom_components directory under the homeassistant config directory:
/root/.homeassistant/custom_components/binary_sensor/rfcac.py:
"""
Binary sensor for 433 interface
Params:
 * name: sensor-name
 * code: 433 code for the sensor (sniff it with rpi_rf_receive or rcswitch
 * type: sensor type (see https://home-assistant.io/components/binary_sensor/ for supported types)
"""
import os
from homeassistant.components.binary_sensor import BinarySensorDevice

def setup_platform(hass, config, add_devices, discovery_info=None):
    """Setup the rfcac binary sensor platform."""
    add_devices([RFcacBinarySensor(config['name'], config['code'], config['type'])])
"""
    add_devices([
        RFcacBinarySensor('TV-Stue', '1234', 'motion'),
        RFcacBinarySensor('Kokken', '123', 'motion'),
    ])
"""
class RFcacBinarySensor(BinarySensorDevice):
    """A RFcac binary sensor."""
    def __init__(self, name, code, sensor_class):
        """Initialize the sensor."""
        self._name = name
        self._state = False
        self._code = str(code)
        self._sensor_type = sensor_class
        self.update()
    @property
    def sensor_class(self):
        """Return the class of this sensor."""
        return self._sensor_type
    @property
    def should_poll(self):
        """Poll for now, to read the file state"""
        return True
    @property
    def name(self):
        """Return the name of the binary sensor."""
        return self._name
    @property
    def is_on(self):
        """Return true if the binary sensor is on."""
        return self._state
    def update(self):
        """Update the state of the sensor"""
        """self._state = False"""
        self._state = os.path.isfile('/tmp/rfcac/' + self._code)
        if self._state:
            os.remove('/tmp/rfcac/' + self._code)

The rfcac modules consumes codes read from /tmp/rfcac/*
It scan this directory for codes (see rescan interval in config file below), and if a code is configured, the sensor is triggered.

rfcac system service
This script uses the rpi_rf_receive command from the rpi_rf module, and places the codes read from rpi_rf_receive into /tmp/rfcac.

Install rpi_rf module using the python installer:
pip3 install rpi-rf

The script: /root/433mhz_receive.sh:
#!/bin/bash
#Create /tmp/rfcac directory
mkdir -p /tmp/rfcac
#Read 433Mhz codes, and append to our rxLog
/usr/bin/rpi-rf_receive &> /tmp/rfcac/rx.log &
#read lines from rx log, and place code into /tmp/rfcac/
tail -f /tmp/rfcac/rx.log |
  while IFS= read -r line
  do
     code=`echo "$line" | cut -d' ' -f6`
    touch /tmp/rfcac/$code
  done

Also remember to make it executable:
chmod +x /root/433mhz_receive.sh

And a systemd unit file, to lauch to on boot (/etc/systemd/system/rfcac.service):
[Unit]
Description=RFcac
After=network.target
[Service]
Type=simple
ExecStart=/root/433mhz_receive.sh
[Install]
WantedBy=multi-user.target


To make in run on boot (remember to run daemon-reload):
systemctl enable rfcac

configuration.yaml:
homeassistant:
  # Name of the location where Home Assistant is running
  name: Vibevaenget 4
  # Location required to calculate the time the sun rises and sets
  latitude: 55.8494015
  longitude: 9.8414076
  # Impacts weather/sunrise data (altitude above sea level in meters)
  elevation: 5
  # metric for Metric, imperial for Imperial
  unit_system: metric
  # Pick yours from here: http://en.wikipedia.org/wiki/List_of_tz_database_time_zones
  time_zone: Europe/Copenhagen
# Show links to resources in log and frontend
#introduction:
# Enables the frontend
frontend:
http:
  # Uncomment this to add a password (recommended!)
   api_password:
# Checks for available updates
updater:
# Discover some devices automatically
#discovery:
# Allows you to issue voice commands from the frontend in enabled browsers
conversation:
# Enables support for tracking state changes over time.
history:
# View all events in a logbook
logbook:
# Track the sun
sun:
zone 1:
  name: home
  latitude: 55.8494015
  longitude: 9.8414076
  radius: 200
  icon: mdi:home
zone 2:
  name: dis
  latitude: 56.069829
  longitude: 9.9784613
  radius: 200
  icon: mdi:factory

# Weather Prediction
sensor 1:
  platform: yr
sensor 2:
#Travel times
#https://maps.googleapis.com/maps/api/distancematrix/json?origins=Vibev%C3%A6nget+4,Horsens|Denmark&destinations=H%C3%B8jbjerg|Denmark&key=AIzaSyCtVwdi82LakzZso0mmmwWdIsB-6AuNRfg
  platform: google_travel_time
  api_key: AIzaSyCtVwdi82LakzZso0mmmwWdIsB-6AuNRfg
  origin: zone.home
  destination: zone.dis
#Hardware Sensors
sensor 3:
  - platform: systemmonitor
    resources:
      - type: disk_use_percent
        arg: /
      - type: disk_use_percent
        arg: /tmp
      - type: memory_free
      - type: memory_use_percent
      - type: processor_use
      - type: last_boot
binary_sensor 1:
   platform: rfcac
   name: Kokken
   code: 4523456
   type: motion
   scan_interval: 1
binary_sensor 2:
   platform: rfcac
   name: TV-Stue
   code: 213411
   type: motion
   scan_interval: 1
binary_sensor 3:
   platform: rfcac
   name: V. Skab
   code: 31441141
   type: motion
   scan_interval: 1

#433
switch:
  platform: rpi_rf
  gpio: 17
  switches:
   #Udelys
    foran_lys:
      #4a
      code_on: 5522769
      code_off: 5522772
    have_lys:
      #4b
      code_on: 5525841
      code_off: 5525844
    garage_lys:
      #4c
      code_on: 5526609
      code_off: 5526612
   #Underlys
    toilet_lys:
      #3a
      code_on: 5326161
      code_off: 5326164
    gang_lys:
      #3b
      code_on: 5329233
      code_off: 5329236

#IP Camera's (https://home-assistant.io/components/camera/)
camera 1:
  - platform: generic
    still_image_url: http://192.168.2.245:8001/snapshot?rand=1481214732238
    name: Foran
    username:
    password:
camera 2:
  - platform: generic
    still_image_url: http://192.168.2.246:8001/snapshot?rand=1481214732238
    name: Garage
    username:
    password:
#Traffic
#camera 3:
#  - platform: generic
#    still_image_url: https://www.google.dk/maps/@55.8488738,9.8445633,10z/data=!5m1!1e1
#Alarm (https://home-assistant.io/components/alarm_control_panel.manual/)
alarm_control_panel:
  platform: manual
  name: Alle
  code:
  disarm_after_trigger: false
  pending_time: 10
  trigger_time: 3600
automation:
  alias: All alarms
  hide_entity: True
  trigger:
    platform: state
    entity_id: binary_sensor.tvstue, binary_sensor.kokken, binary_sensor.rcph_skab
    state: 'on'
  condition:
    condition: or
    conditions:
      - condition: state
        entity_id: alarm_control_panel.alle
        state: 'armed_home'
      - condition: state
        entity_id: alarm_control_panel.alle
        state: 'armed_away'
  action:
    - service: alarm_control_panel.alarm_trigger
      entity_id: alarm_control_panel.alle
#    - service: notify.instapush
#      data:
#        message: "Alarm"
automation 2:
  alias: Alarm notify
  hide_entity: True
  trigger:
    platform: state
    entity_id: alarm_control_panel.alle
  action:
   - service: notify.instapush2
     data:
       message: "Alarm state changed: {{ states.alarm_control_panel.alle.state }}"
notify 1:
  - name: instapush
    platform: instapush
    api_key:
    app_secret:
    event: zoneTriggered
    tracker: zone
notify 2:
  - name: instapush2
    platform: instapush
    api_key:
    app_secret:
    event: msg
    tracker: msg

Troubleshooting

If you intend to use the netdisco module, you'll also need to install gcc. If you forget to install it, the hass-log will show this:
INFO:homeassistant.util.package:Attempting install of netdisco==0.8.1
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-jm9r4b2z/netifaces/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-qtnuyhoe-record/install-record.txt --single-version-externally-managed --compile --home=/tmp/tmpgirlvw2j" failed with error code 1 in /tmp/pip-build-jm9r4b2z/netifaces/

Install gcc:
pacman -S gcc


Secure your communication with HA using SSL

Update 2016-02-20: I can no longer recommend using StartCom SSL certificates, as they are no longer trusted in Chrome (as of version 56) nor in Mozilla and possible others.

(using StartCom's StartSSL - free personal SSL certificate)

While it's important (at least for me) to be able to reach my control panel from outside out home), it's equally important that it's secure (it is after all also our alarm system).

Now, HA does have a guide to setup ssl using Let's encrypt, but the certificates are only valid for 90 days, and I've always used StartCom StartSSL, so here's how to use it (these certificates are valid for 3 years, and free for personal use).

Go to startssl.com, and either login or create an account.
Follow the steps to create a class 1 DV certificate.
Since I'm using linux, I used openssl to generate mine (using the command specified by StartSSL's instructions), and just pasted the content of my "yourname.csr" into their webpage.

After generating the certificate, place the private key file (yourname.key) on the raspberry pi along side the 1__bundle.crt file from the NginxServer.zip file form the downloaded certificate file package from StartSSL:

Here how my config looks:
http:
  # Uncomment this to add a password (recommended!)
   api_password:
   ssl_certificate: /hass/ssl/StartCom_ssl_2016-12-22/2_urup.net_bundle.crt
   ssl_key: /hass/ssl/StartCom_ssl_2016-12-22/yourname.key


Okay, so that should be it. Now you can access you installation securely from https://:8123

Remember to specify https, and not just http.


Note - decrypt the private key:
If you private key is encrypted and requires a passphrase, you'll either need to decrypt to, or start HA manually, so you can enter the passphrase when HA starts.

Here's how to remove the passphrase:
cd /hass/ssl/StartCom_ssl_2016-12-22/
openssl rsa -in yourname.key -out privatekey_decrypted.key

And then in the configuration.yaml file, we should use the now decrypted key:
http:
  # Uncomment this to add a password (recommended!)
   api_password:
   ssl_certificate: /hass/ssl/StartCom_ssl_2016-12-22/2_urup.net.crt
   ssl_key: /hass/ssl/StartCom_ssl_2016-12-22/privatekey_decrypted.key

Note 2 - certificates:
The zip file you download from StartSSL contains several other zip files:
ApacheServer.zip, IISServer.zip, NginxServer.zip and OtherServer.zip.
It would be nice if the included a little readme about whats in these, but they don't, so here's the short version:
I first tried to use the certificate from ApacheServer/2_urup.net.crt.
This actually worked just fine on my laptop (running chrome 54), but when accessing the page form my phone (running android 6.0), it complained "Server certificate not trusted".
A bit of googling and I found out that chrome on pc will automatically build the certificate chain, while chrome on android will not.
The whole problem is that while android trusts StartCom as a root CA, the certificate we're getting is issued by StartSSL DV1 (which is issued by the trusted root CA).

On Chrome/linux it will automatically go up the chain, and try to validate all issuers until it find a trusted CA.

On chrome/android it will not!
We thus need a certificate that also contains the parent certificates until it's trusted by android.

In my case this means I need to include StartSSL DV1.
Luckily this is already included in the downloaded file from StartSSL, and it's the one file thats in NginxServer.zip.

Just on a little side note for Apache users, I read that Apache deprecated the
"SSLCertificateChainFile" in version 2.4.8, so you'll also need to use the full certificate from now on.

Update 2016-02-20: I can no longer recommend using StartCom SSL certificates, as they are no longer trusted in Chrome (as of version 56) nor in Mozilla and possible others.

Conclusion

If you've followed this guide you should now have a running, secure setup, where you can send and receive 433 codes to control power switches.

This post got way longer than anticipated, so setting up and connecting the smoke detectors and other sensors will come in another post.

If you have any questions or comments, please feel free to comment.

In the next part I'm also talk a bit about 433 security (and jamming), which is a problem for any wireless system.

torsdag den 18. juli 2013

Fooling around with gstreamer

So, at work I'm playing with gstreamer, which is a quite powerful, but very confusing thing to use.

Basically you build up a chain of elements, linking your sources to sinks though pads.

gstreamer comes with two handy little command-line tools:

  • gst-launch
    Lets you test your chain (before you write it into your program and compile it, just to see it fail :-/)
  • gst-inspect
    Lets you see the options/parameters of each element.

Two words on gst-inspect:
  1. use gst-inspect without any arguments to see all available elements and gst-inspect to see it's options (capabilities).
  2. You can't always trust the capabilities of source elements, as these may depend on the source (example of untrust worthy source element: v4l2src)
    Your source elements needs to be either paused or playing to see what it offers.

All the elements in your chain have one or more pads, that you can use to link a to other elements, each pad has a set of capabilities are negotiated, to see if they are compatible.

Example just showing a sample videostream on screen:
gst-launch-0.10 videotestsrc ! ximagesink

To specify a width and height, we can pass it through another set of capabilities, forcing the negotiation between the elements to set our desired width and height:
gst-launch-0.10 videotestsrc ! video/x-raw-rgb,width=640,height=480 ! ximagesink

The videotestsrc, as a test source, is very generous of the formats it will output.
ximagesink is not; it only accepts video/x-raw-rgb.

You know this by using gst-inspect:
[urup@c4a012 ~]$ gst-inspect-0.10 ximagesink
Factory Details:
  Long name: Video sink
  Class: Sink/Video
  Description: A standard X based videosink
  Author(s): Julien Moutte
  Rank: secondary (128)

Plugin Details:
  Name: ximagesink
  Description: X11 video output element based on standard Xlib calls
  Filename: /usr/lib/gstreamer-0.10/libgstximagesink.so
  Version: 0.10.36
  License: LGPL
  Source module: gst-plugins-base
  Source release date: 2012-02-20
  Binary package: GStreamer Base Plugins (Archlinux)
  Origin URL: http://www.archlinux.org/

GObject
 +----GstObject
       +----GstElement
             +----GstBaseSink
                   +----GstVideoSink
                         +----GstXImageSink

Implemented Interfaces:
  GstImplementsInterface
  GstNavigation
  GstXOverlay

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw-rgb
              framerate: [ 0/1, 2147483647/1 ]
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]


...

And hence we set the video format accordingly to rgb.

One of the more frequently use elements when working with video is the ffmpegcolorspace one. It provides conversion between different formats.

We could therefore write to above as:
gst-launch-0.10 videotestsrc ! video/x-raw-yuv, width=640, height=480 ! ffmpegcolorspace ! ximagesink

It still works, even though the source format is wrong, because ffmpegcolorspace converts it for us.
note: Whenever possible you should make sure your formats match up, instead of converting them. This is especially true if your on a platform with limited capabilities, or you find your playback is lagging.

If you want to know what formats are being used, you can use the -v argument to gst-launch:
[urup@c4a012 ~]$ gst-launch-0.10 -v videotestsrc ! video/x-raw-yuv, width=640, height=480 ! ffmpegcolorspace ! ximagesink
Setting pipeline to PAUSED ...
/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = video/x-raw-yuv, width=(int)640, height=(int)480, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, format=(fourcc)YUY2, color-matrix=(string)sdtv, chroma-site=(string)mpeg2
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw-yuv, width=(int)640, height=(int)480, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, format=(fourcc)YUY2, color-matrix=(string)sdtv, chroma-site=(string)mpeg2
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw-yuv, width=(int)640, height=(int)480, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, format=(fourcc)YUY2, color-matrix=(string)sdtv, chroma-site=(string)mpeg2
/GstPipeline:pipeline0/GstFFMpegCsp:ffmpegcsp0.GstPad:src: caps = video/x-raw-rgb, bpp=(int)32, depth=(int)24, endianness=(int)4321, red_mask=(int)65280, green_mask=(int)16711680, blue_mask=(int)-16777216, width=(int)640, height=(int)480, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1
/GstPipeline:pipeline0/GstFFMpegCsp:ffmpegcsp0.GstPad:sink: caps = video/x-raw-yuv, width=(int)640, height=(int)480, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, format=(fourcc)YUY2, color-matrix=(string)sdtv, chroma-site=(string)mpeg2
/GstPipeline:pipeline0/GstXImageSink:ximagesink0.GstPad:sink: caps = video/x-raw-rgb, bpp=(int)32, depth=(int)24, endianness=(int)4321, red_mask=(int)65280, green_mask=(int)16711680, blue_mask=(int)-16777216, width=(int)640, height=(int)480, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstXImageSink:ximagesink0: Output window was closed
...

Here you see the source element (videotestsrc) delivers video in yuv format.
The ffmpegcolorspace elements receives video in yuv format (sink), but sends it out in rgb format (src).

Ok, simple enough (once you gets the hang of it).
Now to the point of this whole blogspost...

How to tell what formats (capabilities) are supported of any given source element:

Ok, so gstreamer is a powerful thing, and it's error messages are actually understandable, so you know where go and fix it when it breaks.

First up, as stated before, it needs to be connected and running before you actually know this!

gst-launch-0.10 v4l2src device=/dev/video0 ! video/x-raw-yuv,width=640,height=480,format=\(fourcc\)I420 ! appsink

I'm using a webcam (uvcvideo), that is connected as /dev/video0.

Set the environment variable GST_DEBUG:
export GST_DEBUG=v4l2src:5

You should ofcource replace v4l2src with the element name of your source!

Next, make your terminal windows BIG, and run your pipeline as above.
If you're connected to another device slow a slow line (like serial), I suggest piping the output to a file using "2>".

Now, I'm not going to show your the entire output of this, but somewhere in there, there's this (and alot more like it):
...
0:00:00.137710333 [334m30105 [00m  0x8c05120 [36mDEBUG   [00m [00m             v4l2src gstv4l2src.c:450:gst_v4l2src_negotiate: [00m thiscaps: video/x-raw-yuv, format=(fourcc)YUY2, width=(int)1600, height=(int)1200, interlaced=(boolean)false, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 6/1 }; video/x-raw-yuv, format=(fourcc)YUY2, width=(int)1280, height=(int)960, interlaced=(boolean)false, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 8/1 }; video/x-raw-yuv, format=(fourcc)YUY2, width=(int)1024, height=(int)768, interlaced=(boolean)false, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1 }; video/x-raw-yuv, format=(fourcc)YUY2, width=(int)800, height=(int)600, interlaced=(boolean)false, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 20/1 }; video/x-raw-yuv, format=(fourcc)YUY2, width=(int)640, height=(int)480, interlaced=(boolean)false, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1 }; video/x-raw-yuv, format=(fourcc)YUY2, width=(int)352, height=(int)288, interlaced=(boolean)false, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1 };
...

These are the actual supported formats from the camera!
Note: That before this it may print all the capabilities it think it supports (which is nothing worth really)!

You can also read what formats are used from this log, just look a little lower in the log.

There's also the official gstreamer documentation, which is worth reading:
http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/gst-running.html

tirsdag den 2. oktober 2012

downloading video's from dr.dk

Intro

Some time ago I was contacted by my sister, who lives in Canada, with a request.
She would like to show her kids some danish cartoons.
Quite a lot of these can actually be streamed for free in Denmark  because they are made by Danmarks Radio (DR), and are paid for by TV-tax/media-license in Denmark that nearly everyone has to pay.

It's not really the topic of the entry to explain it, but it shot it goes a little like this:
  • Do you own a tv (does not matter if it's connected)? if yes: PAY
  • Do you own a cellphone with WAP (or a smartphone)? if yes: PAY
  • Do you have access to the internet? if yes: PAY
  • Do you live in a cave, with no friends, no real life? if so: You don't have to pay
    • Exception(s):
      • Do you own a radio? if yes: PAY
Ok, I'm being mean here, and I am actually gladly paying 2200 dkr. pr year (thats 380$ or 295€), because they make some good commercial free TV and radio programs.

You talked about downloading tv-shows, and now you're talking about fee's and taxes?
Yes, sorry.

The thing is that DR is forced, like many others, to only display their content to citizens in Denmark.
Since I live in Denmark I thought I'd just download those shows, and share them with her. Problem solved.

And here is (finally) how I did it.

How DR streams it's media

First up, the link to DR's TV-library is here: http://dr.dk/tv

If you are outside of DK, you might not see all the content.
They too, are using Geo IP'ing - if you want to overcome this, use VPN, or get the direct link to the media-content (or do both for best performance - at least I sometimes have problems streaming netflix movies from the US over VPN (Overplay)).
On a side-note: Yeah. netflix is FINALLY coming to Denmark in October, THANK YOU (and only days after that HBO said they would offer something for the european nordic market as well). Suddenly thay all seem to want to catch up. I just hope it's not overpriced, and a subset of the US version (which I sadly kinda know it will be already).

DR finally uses a flash player (before they used a crappy Windows mediaplayer plugin), so now everyone (windows, mac and linux) can join in and see what they paid for. Sadly this also makes it a little more troublesome to get the direct link.

I've found a show that needs downloding (another request):
and clicking on the show, brings up the player:
and it is indeed flash.


Getting the media link(s)

I've tried several of the downloadhelpers / youtube downloaders but none of them seem to be able to find the links for the media.
Luckily there are other ways.

Using info the the player

The player has a built-in console, you can start by pressing ctrl-k.
It should look something like this:

You need to be very fast on the keys, because it only prints the information in the beginning, and the flash element needs to have focus first (if one could introduce a delay between the page loading and the media starting it would be very helpful).
On the above screenshot, I only got the last lines, but it's enough.

Strike that. Each time it buffers, it prints the info again (see the timestamp):

You'll need the info, so keep it open.

Using the source-code

If you can't get to the console, you can read the info from the site source code + javescript codes sent back and fourth.

You need yourself a proper browser (firefox with firebug, or chrome). Because we need to see what the javascripts are sending back and forth.
Turn on firebug in firefox, or developer tools in chrome to start.
Now open/reload the media link.

In the source code, look for a "rtmp://" link: rtmp://vod-prio2.gss.dr.dk:

In the botton pane, go to Network.
I found the request in there somewhere; can't remember how exactly anymore (and maybe they changed it :-/)

Update: Just search for it

A commenter just noted that you can just search for it, from dr.dk's serach box.
Just include "rtmp" as a search keyword.

This finds a ressourceId page in the search results, with all the relevant information.
Here an example (http://www.dr.dk/handlers/GetResource.ashx?id=1320340):
{"resourceId":1320340,"name":"Forbrydelsen III (9:10) Synstolkning","mediaType":"Video","restrictedToDenmark":false,"downloadable":false,"geofilterId":0,"publish":null,"expire":null,"links":[{"qualityId":3416809,"uri":"rtmp:\/\/vod.dr.dk\/cms\/mp4:CMS\/Resources\/dr.dk\/NETTV\/DR1\/2012\/11\/12642391-87bf-4dee-b35d-5fec187a3cdd\/Forbrydelsen-III--9-10--Synsto_5c68a152fa1b441eb23c932307e9f943.mp4?ID=1320340","linkType":"Streaming","fileType":"mp4","bitrateKbps":1127,"width":854,"height":480},{"qualityId":3416810,"uri":"rtmp:\/\/vod.dr.dk\/cms\/mp4:CMS\/Resources\/dr.dk\/NETTV\/DR1\/2012\/11\/12642391-87bf-4dee-b35d-5fec187a3cdd\/Forbrydelsen-III--9-10--Synsto_5c68a152fa1b441eb23c932307e9f943_1.mp4?ID=1320340","linkType":"Streaming","fileType":"mp4","bitrateKbps":562,"width":640,"height":360},{"qualityId":3416811,"uri":"rtmp:\/\/vod.dr.dk\/cms\/mp4:CMS\/Resources\/dr.dk\/NETTV\/DR1\/2012\/11\/12642391-87bf-4dee-b35d-5fec187a3cdd\/Forbrydelsen-III--9-10--Synsto_5c68a152fa1b441eb23c932307e9f943_2.mp4?ID=1320340","linkType":"Streaming","fileType":"mp4","bitrateKbps":248,"width":416,"height":234},{"qualityId":3416812,"uri":"rtmp:\/\/vod.dr.dk\/cms\/mp4:CMS\/Resources\/dr.dk\/NETTV\/DR1\/2012\/11\/12642391-87bf-4dee-b35d-5fec187a3cdd\/Forbrydelsen-III--9-10--Synsto_5c68a152fa1b441eb23c932307e9f943_3.mp4?ID=1320340","linkType":"Streaming","fileType":"mp4","bitrateKbps":122,"width":416,"height":234}],"chapters":[],"productionNumber":null,"postingGuid":"{b4fa025e-b022-44f2-8df7-03624df6f889}","images":[],"durationInMilliseconds":3525000,"site":null,"broadcastType":"Program","runningOrder":null,"postingTitle":null,"postingTeaser":null}
This shows all the needed info.
Thanks for the update. It is much easier then the ways I found.
Hopefully. they'll keep indexing these GetResource pages :-)

Downloading the video


Using linux/rtmpdump


Next we need a linux machine, or at least rtmpdump:


I've already installed rtmpdump, so in the above, I just run it.
The parameters are:
  • -r  rtmp://vod-prio2.gss.dr.dk/cms
    • This is the server address or url
  • -y mp4:CMS/Resources/dr.dk/NETTV/DR2/2012/09/44dbf1d8-4af3-42f4-8b08-92932922e4c7/aa7a95ca4dc641dcae6be4cd63817ed2_564.mp4?ID=1237625
    • this is the actual stream
  • -o test.mp4
    • This is the desired output filename, in this case test.mp4. Select wantever you want.
  • -V
    • this is optional, and gives you debugging output. Very useful, if it dosen't work.
In the console output above, the -r and -y options are mashed together, but from my example you should be able to see where to spilt it.


It's then start downloading the media...
Once done, it will return you to the prompt again, saying "download complete".

Using windows/vlc

Sorry. I can't find a way to set the playpath in vlc.

What the hell happend to blogspot?

Hmm... long time no see, and I see that blogspot has gotten a new name, design and even decided to rename the address of my blog.

Thanks for doing something I did not ask for, or want :-(

Well to be fair, the old address still works, but it now redirects.
The new design is ok, and who am I to stand in the way of progress (after all I don't use it very often)?
The new name is perhaps the worst of the three. blogger.... what was wrong with blogspot? I liked it.

Ok, I sound like an old man, which I am not, so I', just going to shut up, and move on to what I really needed/wanted to do here.... write something useful for other people, and my own reference.


torsdag den 26. maj 2011

Visual Studio (2008) windows forms - c++ - tab order

So, since I've changed jobs I'm now stuck in VS2008.

And let me clarify something:
I HATE IT!

It is worlds from eclipse which I used before, and in most respects I mean this in a bad way.
The UI is littered with menu/options that don't work, in one view or another.
The dialog boxes (even though they are named the same) change function wether you're running an application or not, or if you've open a specific view/explorer.

To make matters worse, I'm tasked with maintaining an application written in C++ and winforms.
Don't get me wrong here; I like C++, and have worked with it for over 5 years, building linux apps and Qt GUI's.

Take the simple task of defining the tab order in a form; its a complete pain.
I googled the topic, because (as usual) I could not find a option for it in VS.
I found this on msdn: How to: Set the Tab Order on Windows Forms
But surprise, surprise, IT DOES NOT WORK!
I don't have the options/views; Absolutely nowhere can I find "Tab Order".
Designer view? Nope stuck in code view as well, but that's another story.

All hope is not lost though.
Open a file browser, navigate to the .rc file that contains your dialog, or other gui element, and open it using notepad++ or some other editor.

Here's the interresting piece of code from the file I'm working:

/////////////////////////////////////////////////////////////////////////////
//
// Dialog
//

IDD_SIGNIN DIALOG  0, 0, 210, 122
STYLE DS_SETFONT | WS_POPUP | WS_BORDER | WS_SYSMENU
FONT 10, "Microsoft Sans Serif"
BEGIN
    LTEXT           "Indtast Medarbejdernummer",IDC_LBL_LOGINNUMBER,68,18,135,13
    EDITTEXT        IDC_EDB_LOGINNUMBER,68,35,113,12,ES_AUTOHSCROLL | ES_NUMBER
    LTEXT           "Indtast Loginkode",IDC_LBL_LOGINCODE,67,53,136,13
    EDITTEXT        IDC_EDB_LOGINCODE,67,70,113,12,ES_PASSWORD | ES_AUTOHSCROLL | ES_NUMBER
END

There are two Labels (LTEXT), and two Editable text input boxes (EDITTEXT).
Tab order is determined by the order they are listed, so if I wanted to change it, I would simply change the order in which they are listed.
Just note that elements that does not take focus will be skipped, so the two labels does not affect the tab order.
Default focus is the first element in the list that takes focus.

Now save the file, and you're done.

Oh, and watch what happens to VS when it gets focus again.
It says the file has changed, and if you wish to reload it.
If you press "yes" here VS will stop working and get restarted by windows.



How to: Set the Tab Order on Windows Forms

søndag den 6. marts 2011

Linux: Running a command/script after resuming from sleep

I've got myself a Asus Eee Top 1602, which I really want to run linux (gentoo) on.

I've got most of the stuff on it working, both the touchscreen, audio and wifi.

There are two issues left:
 * The volume keys on the front don't work, and
 * after I resume from suspend, the display is dimmed to something like 50%.

Previous issues have been:
 * Getting the wifi to work, and
 * Getting the wifi to work after resume.

Things I haven't looked at yet:
 * Memory card reader

Let me just give the some info to the eee top 1602:
 # lspci
00:00.0 Host bridge: Intel Corporation Mobile 945GME Express Memory Controller Hub (rev 03)
00:02.0 VGA compatible controller: Intel Corporation Mobile 945GME Express Integrated Graphics Controller (rev 03)
00:02.1 Display controller: Intel Corporation Mobile 945GM/GMS/GME, 943/940GML Express Integrated Graphics Controller (rev 03)
00:1b.0 Audio device: Intel Corporation 82801G (ICH7 Family) High Definition Audio Controller (rev 02)
00:1c.0 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 1 (rev 02)
00:1c.1 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 2 (rev 02)
00:1c.3 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 4 (rev 02)
00:1d.0 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #1 (rev 02)
00:1d.1 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #2 (rev 02)
00:1d.2 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #3 (rev 02)
00:1d.3 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #4 (rev 02)
00:1d.7 USB Controller: Intel Corporation 82801G (ICH7 Family) USB2 EHCI Controller (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev e2)
00:1f.0 ISA bridge: Intel Corporation 82801GBM (ICH7-M) LPC Interface Bridge (rev 02)
00:1f.2 IDE interface: Intel Corporation 82801GBM/GHM (ICH7 Family) SATA IDE Controller (rev 02)
00:1f.3 SMBus: Intel Corporation 82801G (ICH7 Family) SMBus Controller (rev 02)
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 02)
02:00.0 Network controller: RaLink RT2860
# lsusb
Bus 001 Device 005: ID 0bda:0158 Realtek Semiconductor Corp. Mass Storage Device
Bus 001 Device 006: ID 0458:7063 KYE Systems Corp. (Mouse Systems)
Bus 001 Device 003: ID 05e3:0608 Genesys Logic, Inc. USB-2.0 4-Port HUB
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 005 Device 002: ID 1bfd:1688 TouchPack Resistive Touch Screen
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 004 Device 002: ID 04f2:0402 Chicony Electronics Co., Ltd
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub


Getting wifi to work
I'm using the staging driver in the kernel.
It's under: Device Drivers -> Staging drivers ->  Ralink 2860/3090 wireless support
You can compile it as a module or built it into the kernel, it dosen't matter.
What matters, is that you go go Ralink's website, and download the firmware file rt2860.bin, otherwise you'll just get something like this in dmesg when you "up" the wifi:
eeeTop kernel: [ 1321.424360] rt2860 0000:02:00.0: firmware file rt2860.bin request failed (-2)
eeeTop kernel: [ 1321.424378] ERROR! NICLoadFirmware failed, Status[=0x00000001]
eeeTop kernel: [ 1321.424397] rt28xx Initialized fail!

You'll find that here: http://www.ralinktech.com/support.php?s=2 and you'll need to get the PCI firmware of course.
Download that, extract, and place it under /lib/firmware (you might have to create the directory).

After this, I just emerged wpa_supplicant, and I'm able to connect to my wifi (WPA2 protected).

Getting wifi to work after resume
I found out that after the machine has been put in suspend, and reawaken, the wifi does work reconnect.
wpa_supplicant still reports the wifi as connected (with an IP adress and all that, but there's no connection).
I've read some people have some success by disabling some kernel options (specifically wireless extensions), but that  did not work for me.
Instead 'm just restarting the wifi after the machine resumes.
First I thought of using acpid, but I could not find see that that reports any suspend/resume events.
Anyway acpi is old, and maybe deprecated, now with hal (which is also deprecated) and policykit/consolekit, the last one being the future.
I've moved the system completely over to consolekit/policykit, and it work 100% now. (power management, shutdown, suspend, hibernate). But how to migrate was not that well documented, but thats a different story.

I found the solution to be pm-utils.
Even though it is written for HAL, it seems to work for consolekit as well.
Best of all, it is very easy to setup.
Under /etc/pm/ you'll find the configuration.

Under /etc/pm/sleep.d/ I just made a new file 50wifi, and put this in it:
#!/bin/bash
case $1 in
   hibernate)
       echo "Hibernating"
       ;;
   suspend)
       echo "Suspending"
       ;;
   thaw)
       echo "Returning from hibernation"
       /etc/init.d/net.wlan0 restart
       ;;
   resume)
       echo "Returning from suspend"
       /etc/init.d/net.wlan0 restart
       ;;
esac

Next, just make the file executable:
# chmod +x /etc/pm/sleep.d/50wifi

And thats it.
Now, every time the computer comes back from suspend or hibernation it restarts my wireless, and my connection is back within seconds.

I might update the blog, once I the re two remaining issues resolved.
I can live without the volume keys, but the dimmed display really needs to be fixed.