RDM Responder Tests are Ready

[nomis52]

I've completed most of the work on the E1.20 responder tests now. As you can see from the output below the test suite has 162 tests as well as a bunch of checks for warning and advisory messages. In writing these tests I've found all sorts of cool ways that responders break and/or output garbage .

----------------- By Category -----------------
                    Control:    8 /  8     100%
                Sub Devices:    1 /  1     100%
              Configuration:    9 /  9     100%
        Product Information:   17 / 17     100%
         Network Management:    4 /  4     100%
         Core Functionality:    2 /  2     100%
           Error Conditions:   86 / 86     100%
           Display Settings:    4 /  4     100%
               DMX512 Setup:    9 /  9     100%
      Power / Lamp Settings:   12 / 12     100%
                    Sensors:    6 /  6     100%
          Status Collection:    2 /  2     100%
-----------------------------------------------
160 / 162 tests run, 160 passed, 0 failed, 0 broken

Sometime in the future I'll start adding the E1.37 PIDs. In the meantime if anyone wants to run them for themselves information is here: http://opendmx.net/index.php/RDM_Responder_Testing

Now here's the challenge for everyone: I haven't found a device yet that can pass every test. The output above is from my software emulated responder.

Simon

[mike_k]

Originally Posted by nomis52

I haven't found a device yet that can pass every test.

One interesting question, is it considered "not passing" if the responder is forgiving? For instance, not NACKing a GET:DEVICE_LABEL with a PDL > 0?

[nomis52]

Responding with an ACK where too much data is provided (like the example you provided) will result in a warning but not a test failure. Responding to a PID that requires data with an ACK generates a test failure.

The reasoning is this: while many would argue that responders should be liberal in what the accept, this can be taken too far. The problems that we face today with browser standards are an example of this, if the early browsers had been much stricter in parsing html we wouldn't be in the mess that we're in.

By being strict, responders can expose bugs in controllers that would otherwise go unnoticed. It's particularly bad in that if you have a liberal responder & buggy controller, introducing a strict controller into the rig will cause people to blame the wrong device.

If RDM were further along I'd have a different opinion but given how early we are in adoption, I want to try and enforce strictness now. Ideally a device would have a custom PID ENABLE_STRICT_MODE (default:True) which would control the level of strictness.

Anyway, that's enough from me. We do have the first responder that passes all tests now - a Creative Lighting Slammo dimmer : http://creativelighting.com.au/index...d=13&Itemid=31