DS1307 Real Time Clock hacking: the emulation

Maybe some of you knows well the so called GertDuino, an expansion board of the RaspberryPi. It is like an Arduino which can communicate with the Rasp, and moreover it also mount an Atmega48 which can be used to emulate an RTC. Yes, I’m going to talk about this.

The challenge: proof-of-concept of commercial real time clocks emulation (DS1307).

I saw my board with a PIC18 microcontroller, that embeds an RTC module inside the microcontroller. Since I’ve recently tried the on-chip RTC capabilities of my PIC, the idea is born spontaneously: can I cheat my Pi by making it see a fake (but working) DS1307 RTC? Well, if I’m writing this, means that the answer is yes. And this, of course, can be applied to any single board computer (like Raspberry) which carry an I2C interface. This can be helpful even if you have a different microcontroller, because the software libraries can (in an ugly way) be portable! Oh, and the microcontroller which emulates the DS runs at 3V3 and it is even 5V tolerant, so no need of any adapter, when working with RaspberryPi.

The RTC nostalgia

ds286Dallas Semiconductor were quite known as a company which design and manufacture mixed-signal semicoductors and a lot of digital and analog complete amazing modules (like voltage references, battery handlung, real time clocks ecc). With our hands we can touch the old real time clocks inside old computers, like the firsts IBM in the Intel 80286 epoch. A lot of sweet memories with my old and still working IBM PS/2, with the (now battery drained) DS1287 (photo on the left).

Now the company has been acquired from the Maxim Semiconductor, which still produce some of their products. The RTC series still exist and will continue, as long as the driver support from embedded Linux distributions, and are currently still used in a lot of microcontroller based projects and commercial products. The firsts embedded RTCs have their own battery embedded in the package, which is called E-DIP package. They was huge and if the battery wears out, you can forget to disconnect the supply, or change the chip, or… breaking it and apply an external battery!

Apparently, a famous RTC used, especially in hobbysts and makers is the DS1307. A very basic RTC, without any alarm system capabilities. For that reason, since my microcontroller with RTC module is supporting alarm clocks too, I may need, in the future, to change and emulate a different RTC, like an DS1337. Ah, today are packed in standard DIP or small SO packages without the internal battery, but I’ve found that are still exist the E-DIP packages. Interesting.

The DS1307 works with a logical registers interface. Physically the access is made through 3 wires using the I2C bus, where the RTC is always a slave (SCL, SDA, GND). Except for the first byte on the bus, which identify the slave (here this DS has always a 0x68 bus address) plus a read/write bit mode, the real data, which can be named as payload, can be sent in writing (to DS) or retrieved in reading (from DS) mode. Refer to datasheet for details. The trick is simply to emulate the behaviour and timings. If you know it, you can become it. 😀

Firmware and code portability

When I started writing the code for testing an EEPROM, since the need of very standard interface like the I2C, I thought to use the already written libraries from the XC8 compiler: but I found their names and usage too less intuitive and with the same time spent in learning how to use them I could write my own. So as a good exercise I wrote them and now, for this project, are also improved a bit, using a lot of defines and enumerations, allowing to increase the functionalities and simplifying their usage and writing, and keeping them quite inexpensive in terms of allocated resources. The tool used are:

  • MPLAB X IDE, with the XC8 compiler (free version)
  • PIC18F47J53, mounted on this italian development board, known as PIERIN Pic18.
  • 32768Hz quartz, disassembled from a thermostat or so, I didn’t remember
  • RaspberryPi for tests
  • GitHub
  • Coffee

I’ve also though that if a new microcontroller needs to be changed, the code should be portable, or at least should not be painful when porting it. So I thought to a very primitive level of abstraction layers, written in pure C. The whole code is here on GitHub.

Abstraction Layers

The firmware is composed by three logical structures, but keep in mind that I’m talking about firmware, so are not quite imponent and distant these layers between each other, as in common PC applications:

  1. The classic main and other configuration files, like interrupt, config bits ecc, that are common to other functions, not only to the DS emulation in case other tasks are implemented. Let’s call them as common layer, because is common to the other layers, it is a “vertical” layer.
  2. The implementation of the higher level of the DS algorithm, that contains procedures which are calling the hardware drivers in a proper order. Name it as api layer (a very basic form of application programming interface). More functions at this level means more things done by the MCU. Here can be managed the hardware allocation for each task.
  3. The drivers called and used by the previous layer, in which are present the procedures that are implementation dependent (the hardware handling). Changing the hardware means change (or add) some row in these files. So they are a basic form of hal layer (hardware abstraction layer interface).

Each of them are interacting with each other, roughly, in this way:

fw_layers

Thanks to the API layer, I can improve the emulator with more features and other hardware support, due to practical experimentation purposes, leading to a more enriched, open project. For sure I will need to implement a software RTC feature for platforms that don’t have that module. Last but not least, the “compass” of the emulator was this table, which is the register interface of the DS1307, one eye on the code and the other on this table:

ds

The various defines in the code in the HAL and API layers are related to this table. Those definitions are not well compartmentalized from one layer to another, because I thought to be more flexible w.r.t. emulator platform rather than the emulated object itself. Nevertheless, a new emulated object will bring to a complete review of the layers, because different APIs are needed as much as a different hadware (the HAL) behavior as different are the required procedures of the driver.

Hardware setup and demo tests

Now it’s time to test if everything works. I started directly on the RaspberryPi using the hwclock routines from terminal and its embedded I2C module. Strange things happened, from writing an year and reading another to get the Pi freaking out due to detected error configurations on the DS. So more simple functions were needed to test it, in a very fast way. Here my old school Arduino Duemilanove became very useful, by using and adapting some DS1307 libraries, to run a complete test of basic interactions, check the change of settings and play with the NVRAM.

Coherence and settings tests

Few main bugs in my code are find by using some Arduino libraries. But I used them to see the first heart beat. The Adafruit RTClib, where some of them are modified to fit my first alive tests. Then a code from a guy on Instructables, I’ve just googled for a testing function. Taken the I2C pins A4 for the SDA and A5 for the SCK from the Arduino, and connecting them to my Pierin, I finally make the PIC18 seen by the Arduino. But it was frozen. Of course, I forgot the little brother, the timekeeping crystal on RC0 and RC1 oscillator pins.

32kcry

And then, it started breathing:

alive

The doctor said – time of birth: 12pm, 48′ and 53″. The date was taken using the RTC.adjust(DateTime(__DATE__, __TIME__)); call. Moreover, I can easily test the output pin functionalities, halting the clock, change output settings and  testing the NVRAM by using the RTClib. At this stage, the difference with the original DS1307, is the frequency at the output of RTC pin. It is not a bug, but the PIC hardware support only 2 modes instead of 4. If needed, can be done easily in software, but now, as stated in the first title, I’m at a proof-of-concept phase. Below there is the output pin table of DS1307 with shaded modes that are not currently available in this emulated module:

csmode

RaspberryPi: connection and setup

The systems which will use this emulator would be also any computer with an I2C port. Here, a Raspberry Pi is very appropriate. I’ve the RPi B rev.2, so I’ve used the I2C module 1, using pins 2 and 3 (plus GND and 3V3) of the R2 reference. Connections are made by taking care, in testing phase with PIC18 powered by my PC, to have the ground in common with all systems and then connecting the SCL1 and SDA1 ports of the PIC to the respectively GPIO2 and 3 of the Pi:

Now it is needed the software setup under  Linux (here Raspbian). One can follow the standard steps for the DS1307 written everywhere, but sometimes bad things happens. Here my final setup (I would to thanks the source http://en.gnublin.org/index.php/RTC_DS1307).

    1. Firstly check if the RTC bus is alive, by typing sudo i2cdetect -y 1: detect
    2. Then enter in bash mode by typing sudo bash and only then type echo ds1307 0x68 > /sys/bus/i2c/devices/i2c-1/new_device to put on the i2c bus 1 a new device called ds1307 at address 0x68. This data is being used by the Hwclock procedures. If an error arise try to remove something, like echo 0x68 > /sys/bus/i2c/devices/i2c-1/delete_device and repeat the new device insertion.
    3. Load the already present (it should be) module “ds1307” to the kernel using (I did it in bash mode but it is not a mandatory) modprobe rtc_ds1307 2>/dev/null 1>&2. The numbers are meaning that the standard error (2) is trashed and write the output (1) also in in stadard error. This allow the hwclock utility to avoid problem related to “IOCTL” not working calls in which you can step into, and it is used to keep trace of the output of commands without printing them on the screen.
    4. ONLY IF YOU GET ERRORS WITH HWCLOCK: To avoid further problems, which are not taken into accout by the standard guides, it is needed to add the physical rtc device, which often is not present, allowing problems of ioctl type. I’ve added to rtc0 by typing: mknod /dev/rtc0 c 253 0 2>/dev/null 1>&2.
    5. Check with dmesg that is something positive like this:
      [ 5061.101787] rtc-ds1307 1-0068: rtc core: registered ds1307 as rtc0
      [ 5061.101837] rtc-ds1307 1-0068: 56 bytes nvram
      [ 5061.101895] i2c i2c-1: new_device: Instantiated device ds1307 at 0x68
    6. For verification purposes, typing lsmod should output the rtc_ds1307 module somwehere.

Now exit from bash, and if you want you can setup a script from the linked Gnublin source. I didn’t test it. Now, if you need to test the device using the i2c utilities, you need to delete the device (step 2) to free the bus address reservation. There is not much to show on photos, but I put my test bench just to make this article a bit more coloured. 🙂

setuprtc

And the tick is got.

gottick

Unix epoch and misunderstandings

A curious thing is related to the date previous to year 2000. If I have the 1970 as starting date (the first year of the Unix epoch), issuing the hwclock -w will write this date to the RTC. But any read was wrong, giving back the year 2066. Writing manually the registers produces a normal results, so the only conclusion, since these RTC are supporting dates from years 00 (2000) to 99 (2099), the hwclock is making something that is misunderstood since will work on more than 8bit to represent that value, but the RTC understand this as a 2066 due to some millennim transformation. In fact, with date from 2000 to 2038 everithing is fine and surprisingly everything it is not my fault. But why 2038? Don’t get me wrong, the RTC module can handle dates to 2099 correctly, but any Unix derived OS can not. In fact, I’ve sperimented that dates (dd/mm/aaaa format) until the 19/01/2038 03.14.08 am are correct. But the second later the end of the Unix epoch, any read will indroduce a mismatch error. And any set over this date is notified to be… correctly wrong. All this is because the timebase variable which keep tracks of the total seconds from the 01/01/1970 12.00.00 midnight, is a 32bit signed and will wrap at that date in 2038. That will be the thirtyennium bug or whatever, but I doubt that the world will collapse in a black hole. Here, is what an end of epoch will appear:

unixwrap

If you wake up in the morning, without smog, smarthphones, computers and in the street you see only stagecoaches and horses with gentleman or a huge poverty, well, probably you have travelled in time. In the meantime, the others will see on their PC/smartphone/ecc clocks this awkward date. There are many proposed solutions, but it is not a trivial thing.

Future improvements and lessons learned

During this trip I’ve discovered how powerful can be an on-chip module, how much can be simplified the programmer’s life and firmware execution. Since I’m an electronic engineer at heart, I’ve made some progress in the coding art, despite it still look not so perfect. And with the excuse of debugging the I2C bus, I’ve finally bought the PicKit 3 and my life became easy and free from bootloaders or fuse registers unchangeable.

As soon as I can I’ll try to implement the alarm part, which can be used to trigger stuff on the GPIOs of the RPi. In this way I may need to became compatible with another version of Dallas Semiconductor real-time clock.

And now, getting involved in bus protocols and by discovering how powerful can be just an 8-bit microcontroller, how is difficult to fill even 5KB of flash and with the help of past experieces, new ideas are under development. As an engineering mind is in constant motion.

Follows DS1307 RTC hacking part 2 HERE.

Il caos è l’anagramma di caso?

320px-Streichholz

Ho un’idea che a stento trattengo da un po’ di tempo. E’ quella di pensare che la matematica che costruisce ogni cosa sia ancora fortemente limitata. L’infinito è una grandezza che viene esclusa a priori e in qualche modo aggirata per poter ottenere quel che si cerca. Per questo la matematica è potente, ma limitata al tempo stesso. Per definizione non può spiegare l’evolversi dell’universo nel suo dettaglio più piccolo.

Per esprimermi meglio, pensate a cosa vedete quando accendete un fiammifero: è sempre una fiamma, con del fumo, ma i suoi movimenti sono sempre diversi e mai uguali o ripetuti in ogni dettaglio. Così come i più classici fiocchi di neve. Se si applica la matematica ad una scienza, si deve guardare alla statistica, che altro non è che una forma della matematica per determinare eventi influenzati da un numero di variabili che è praticamente (se non realmente) infinito. Per questo la statistica non è certezza. Per questo i risultati della matematica, nella realtà, sono affetti da un’incertezza.

In realtà se progetto un circuito elettronico, o costruisco un qualsiasi oggetto, esso avrà delle incertezze di costruzione che vanno considerate come limite per capire che più preciso di così non posso. Questo perché in qualsiasi cosa premeditata, una quantità di incertezza c’è sempre, dal momento che l’universo è intrinsicamente caotico. La matematica nella realtà ha quindi sempre un briciolo di statistica, mentre la matematica in sé è ideale, come il ragionamento di una persona. Ha solo il limite di non concepire l’infinito, come il ragionamento di una persona.

Se si potesse trattare l’infinito, lorentz-artil caos non sarebbe più considerato tale. Se in ogni istante di tempo infinitesimo venissi a conoscenza di tutte le interazioni di tutta la materia ed energia dell’universo tra ogni singola infinitesima particella e quantità di energia, saprei spiegare perché, nell’istante di tempo successivo, quell’evento si svolge proprio in quel modo (vedi il fumo di un fiammifero che cambia sempre direzione).  Il problema è che c’è il rischio che le particelle siano infinite, irrangiungibili e per di più ho bisogno di quantità infinitesime: cioè infinitamente piccole, come dividere un numero per infinito. Questa divisione porta a zero come risultato, ma così il mio istante di tempo resterebbe sempre fermo, mentre io voglio in qualche modo avanzare per capire la soluzione di questa “equazione a parole”. E’ questo il limite della matematica di oggi. Inoltre, la cosa sorprendente, è che allo stesso modo di come si potrebbe avanzare in modo infinitesimo, si potrebbe anche indietreggiare. Il tempo sarebbe davvero considerabile come una qualsiasi altra dimensione.

Insomma, appena si parla di infinito, come le possibilità che porta il caos, la matematica cede. Se fosse così evoluta da poter gestire in qualche modo l’infinito, si potrebbe rispondere davvero a tutte le domande, specialmente a quelle fantastiche di un bimbo che ad ogni risposta aggiunge sempre un “Perché?”.

Eppure, tra 22 e 23 esistono infiniti numeri decimali, ma la colonnina del mio termometro a mercuirio si alza da 22°C a 23°C in un tempo finito, ma con una incertezza di misura.

Eppure, il caos ci circonda, e ci prende in giro.

«L’infinito attuale si presenta in tre contesti: in primo luogo quando si realizza nella forma più completa, in un’essenza mistica completamente indipendente, in Dio, che io chiamo Infinito Assoluto o, semplicemente, Assoluto; in secondo luogo quando si realizza nel mondo contingente, creato; in terzo luogo quando la mente lo coglie in abstracto come una grandezza, un numero o un tipo di ordine matematico.»

cit. Georg Cantor

The classic hardware hacking: ATX power supply

main

If you like playing with lab instruments, enjoy to make things and design electronic stuff, you are probably attracted from the idea of create a new powerful (with some limitations) bench power supply by using an old ATX supply. Moreover, I need an high power supply to test my motors. For sure, building your own (not switching like this one) is possible and far more instructive.

WARNING

You should follow this and/or other guides related to high voltages only if you know what you are doing. It is also a good thing to know the Ohm’s law and dimensioning a resistor to light up an LED. Remember that inside the PSU there are high voltages, potentially deadly. Despite I’ll not try to manipulate these voltages, the insulation integrity must be kept. Do this hack at your own risk! See also the disclaimer.

###

Let’s start

Firstly, in case it is not clear, you need to find an old ATX power supply of a PC that you don’t use anymore. Usually it’s always a low cost supply, those that will never reach powers that are mant to be supported. Those sold to be, for example, 300W can output more than 200W nicely, with a lot of voltage combinations but with some drawbacks: without adding some hardware, are neither voltage nor current adjustable. This means that if you short something on your low power PCB, you can destroy it because it will source all its possible current. This is a situation that a bench born power supply will avoid.

TIP: when you buy an ATX power supply of a “generic” chinese sub-brand, apply a security factor of 0.5 when planning the power budget of your PC, usually they can stand a stable current/voltage which is usually far less than what is written.

Then let’s open it and see what’s inside. This one is a “chinese” one, with no EMI filters nor a power factor corrector.  Considering that my version of hacking will have all cables tied inside, that can arise an heat problem which is ignored right now, because can be easily solver later, after the completion of the bench supply.

Here you can see the missing filters, along the uncomfortable soldered fuse:

no_emi

But never mind, I continue, aware that this power supply can be not compliant with some RF constraints. If the final job is accomplished, I’m are already happy.

I cut out all the connectors and applied as many holes on the chassis as I need (see the bottom of this article). I’ve considered 3 indicators (+5VSB orange LED, 230VAC orange neon lamp and a green Power Good LED indicator). Other holes should be dedicated to the voltages I want to obtain. I used a normal drill, so was quite difficult to obtain the holes. Everyone must sadly arrange with what is present at home. But just rember that there are so many ways to implement your own implementation, and mine can be done more cleverly than what I’ve realized at the end.

Connectors and signals

Since those power regulators are very complex, I’m limiting to check the compliance with the ATX standard and follow that to manage the few control signals. For that same safety reason, this “hacking” does not involve any modification to the original PCB.

The ATX (version 1 and 2) follow the ATX standard to define control signals and some internal regulations. At the www.formfactors.org can be found the full specification of the connectors that we’re facing to. Mine is version one, you can check by googling or referring to ATX standard to understand what version you have in your hands.

connATX

I’ve to identify all common voltages, like +-5V, +-12V, 3V3, COM ecc and cut out the connectors. Then merged all the same voltages togheter. Since the GNDs wires are too much, I’ve split them in two separate threads of wires. I’ve joined the cables with the same voltage and appllied some solder tin to make the pin of merged cable stronger. Then I’ve soldered them to the crimp connector which will be inserted to the back of banana connector that I used (below the crimp is pointed by the black arrow), and closed everything in a thermo shrinking rubber, if the crimp does not already have some “protection”. This gives a mechanical strenght and electrical insulation. It looks not perfect, but it’s very solid and easy to mount. The tin on the bolt has no meaning, as you may see there are few signs of experimentation on alternative fixing leading to melting some plastic, but neglect them.

merged

Remember, depending on what you want to obtain, to keep apart the VIOLET cable (+5VSB), the GREEN and GREY and one +5V RED cable. As one can easily check holding the original connector, those are the voltage of Stand-By, +5VSB, used to power up some peripheral of your PC like the PS2 keyboard, some of USB connectors and the auxiliary functions on the motheboard. See:

atx_connReading the specifications of the ATX, the Green and the Grey are used to provide a digital TTL signal on the PS_ON#, being low to start the supply and can left float, by defined by the standard, to assert it high (if the PSU is designed properly, otherwise you can use a 10KΩ pull-up resistor, here is not needed though), then, when voltages are considered stable, an high TTL signal is readable from the PWR_OK. But if I want to use this signal to light up an LED I need a transistor. Here I will use a switch to turn on the system, which will tie the GREEN cable to GND and I will use a BC337 bjt to receive the OK signal to drive the LED, because the TTL port can sink up to 4mA which are too few to bright an LED correctly. Moreover, since I want to light-up the LED when the signal is high, the TTL port can source a max current of 200uA with a 1KΩ  resistance: a bjt is totally needed.

A FACT: The LED will turn on very dimly without using the BJT, so maybe more than 200uA are provided and the TTL can go lower than 5V. Even using a BJT can absorb more than 200uA to keep the transistor saturated; it could work in the active region, but I don’t like it when it is needed just to visualize the presence of a digital status. Ideally could be a MOSFET, but I’ve no one which works at a signal level. What can I do?

I’ve seen that LED has turn on dimly because there was at least 1mA.  Digging in the circuit, I discovered that this is the output of an LM393 comparator: it can sink only, in other words the logic high signal is provided by a simple pullup resistor.

Once all connections of the schematic below are made “on the air” I’ve closed every thing in the magic shrinking rubber as it’s reported on images below.

One thing on dimensioning the circuit:

The power dissipated by BJT is devised by multiplying the collector current by the voltage drop. Here Ic=15mA (see Icsat formula below) and Vcesat = 0.7V; so P = Ic*Vcesat = 10mW and to be safe we can consider it at least 3 times, P = 30mW. Datasheet says that the TO-92 package can dissipate 200°C/W and provide a graphical representation, with a maximum temperature of 150°C. Yes, at 25°C I can dissipate more than 600mW, but it will burn if you turn off your air conditioner: what are my ranges? 30mW means an increase of 6°C, which means that the power supply can stay lower 144°C. If I will get to that temperature, the problem is somewhere else. So everything is good.

Of course I want to use the BJT in saturation, so to keep the BC337 Vce under the saturation value, I need to provide some high base current. The Vcesat = 0.7 V by datasheet, obtaining a saturation current of Icsat = (5V – 0.7V – 1V)/(220Ω) = 15 mA, where 220Ω is the LED resistor and the LED drops by 1V the voltage. To devise the base current we apply an overdrive factor of 10, obtaining a beta = 100/10 = 10, where saturation beta is 100. This lead to a base resistor of (5 – 0.7)V/1.5mA <= 2533 Ω. The overdrive will cover the BJT approximations. I put a 1.8KΩ resistor in the base, added to the 680Ω already mountend to be the pullup of the comparator.

Good. And in fact, the LED looks pretty bright. A way to think: the current should be limited by the resistor and not limited by the BJT, when used in saturation.  🙂

See images below.

bc
bc_holdings

While the monitoring of the +5VSB can be made simply attaching an LED (orange here) to see if is present, just a series resistor is put (allowing some mA, like 10/15mA). But, be aware to create a fork of the Violet cable, since here there is only one wire and we want also to use this voltage as output for the bench supply! As shown below. The arrow indicates the LED connected to the blue cable derived from the violet (see the fork from violet to blue on the left image), and the series resistor is hidden under some shrinking rubber (not visible here).

Due to lack of dimension of holdings dedicated to LEDs (because I used a drill which was not the proper one to be used to make holes on chassis), those LEDs can’t be fixed mechanically in the holes. So I just glue them on the respective holes. You can now insert the banana connectors in the other holes and screw in the bolt to keep the crimp connector (the one indicated before with the black arrow) strongly attached to the back of banana ones. I also directly connected to the 230VAC an orange neon bulb to see always if there is AC line inserted. Insulated with a proper not home-made holder.

Things to keep in mind

Oh yes! There is a remaining one +5V cable. This is used to power up some dummy resistors. If you search in the web, you can find the general advice to use a load resistor of 5W or 10W of 10Ω. I’m not always agree with that.

Depending on the power supply used, you may need to power a dummy load to the 12V or 3.3V. Usually for old ATX power supply like mine the 5V should be loaded. Here I’m referring to this voltage, but situation can vary with other PSUs.

The resistor is used to let flow some current from the +5V rail and close the loop of voltage regulation feedback, which is applied the the +5V also to regulate the other rails (3V3, 12V ecc). Since I don’t want to waste energy and generate uselss heat in the resistor, I measured the stability and the absolute value of voltages by varying the resistor. I came up with a minimum current which is arount 100mA that seems sufficient to keep the supply stable. So I used 4 resistors of about 38Ω to obtain a final equivalent resistor of the same value but able to support twice the power of the single component (2 series pair of resistors, each with a parallel of two single resistors). I also check by removing the PCB if the grounding was directly connected to the chassis and it is. So the chassis screws can be used to close the circuit. The space is precious here. Also, the resistors are not quite hot, so it’s ok. The white tape in the picture has insulation purposes, underneath wires are soldered, of course.

dummy_res

Final result

I put some effort to keep the airflow as clean as possible, the heatsink on the left has also a couple of wings that are bended to not touch cables too heavily. For now, seems that there are no heating issues. All voltages are good  enough excepts for negative ones: could be my fault, but they were few mV under the nominal value also when the power supply was used in a real PC, prior to any modification. The technical reason is that this power supply is a crappy one. The whole result is posted here below.

wires

DSCN3931

lit_up

A clever thing could be use the hole on the front to hold the switch. I didn’t, I’ve complicated my life uselessly, cutting a piece of metal (the one on the relief) to give a shape on the space where the switch slides, because the rectangular shaped hole was not perfect at all. But holes for screws were quite nice and the switch is fixed strongly on the chassis.

What I’ve learned

As you may notice, I wanted to have a power supply which was blue. So I spent a week trying to painting it and learning all steps to achieve a durable paint on a metal chassis. I’ve also tried to paint some letters on the top to make them visible inside the blue main paint, just for experimenting some painting techniques. I used the primer to allow me use the spray can to cover the whole chassis with this blue that you can see. And it’s not perfect at all the whole result. But I’ve never painted a chassis before and I feel satisfied.

The whole hacking, at the end, has taught me basics about painting. Yes, agree, WTF! 🙂

Blink an LED over the internet: the begginer’s low level approach

ledoth

In this article we continue our road to the IoT approached at a low level, or better, at bottom-up approach, that will be applied in the future to my UART interpreter. Here I will talk about how to drive an LED over the Internet. So as always I’m using Python libraries to get things done safely, quickly, but also make them useful under the knowledge point of view. So, if you are trying to do similar things, this can be an article that is not completely useless. And remember, if you can blink an LED, you can do whatever you want… This is the “Hello World!” of the basic IoT. Maybe even less.

But be aware, there are a lot of guides that allow you to do the job quickly. You are reading this because I am developing an hardware platform which works differently from most of ready-to-go solutions, and I usually need to develop my own application and if something become useful, I write about that, like in this article.

Let’s start

As a starting point, you have to understand what you need. Here, since I don’t have any ethernet/wifi adapter to be used by an MCU and considering that I may need some calculus power, I used the well trusted RaspberryPi B. I set it up with a basic Raspbian distro and configured it to use a static IP address local to my LAN. Forwarding the traffic from outside is made by the router connected to the internet.

If you start from scratch and don’t have the DHCP disabled and the RPi doesn’t have a fixed IP, you can connect it to the LAN and observe by looking at the router at what address is connected to. Then to access to the board once you know the IP, you should have the ssh daemon running, this is the default normally. In Windows, I use PuTTY to access to it.

Once entered in our RaspberryPi at its new address, we set a static IPv4 address (note that in the so called “IoT” stuff the IPv6 should be used, but for now it is not a big problem) to avoid checking the IP every times. Open the file /etc/network/interfaces, it should look like this:

auto lo
iface lo inet loopback
#uncomment to change loop address
#address <address>
#netmask <address>

auto eth0
iface eth0 inet static
address 192.168.1.5
netmask 255.255.255.0
#uncomment for use RPi as router
#network <address>
#broadcast <address>
gateway 192.168.1.1

where iface eth0 inet static matters, because in this way we assign a static local IP address w/out using the DHCP of the router. Remember to set the IP accordingly to the gateway, where in this case represents the address of my router. The netmask should be identical.

Restart the network typing on the console:

sudo /etc/init.d/networking restart

and may not work, in that case simply reboot the RPi. Now, every time you boot the Raspberry and connect it to the router, it should always have the address that you have choosen, in this example is the 192.168.1.5

The scripts

As usually, I wrote a script that should run on the RaspberryPi, called server and another client that you can run on whatever PC, smartphone which can interpret python commands. Those are derived from the same functions described in this article, and with a new one, fine_data(), to refine the data received by cutting out the number of byte attached at the head of the string prior to send the data. Be aware that the server implemented is not working like a real server, it is capable to handle only one connection at a time, because my application because I want it. It’s a kind of  be-or-not-to-be-server. Otherwise, the basic concept is that you need to put the server.accept() call in a loop and exploit multithreading. That is why servers are needing high parallelism and memory to allow parallel execution, instead of single thread performances.

Both scripts uses an infinite loop to interpret the data, and the client send to server whatever command is inserted from the user. The server itself should verifiy the command and decide what to do. So what’s new, now, is in the server, where thanks to the library GPIO, prepare and drive one pin of the RaspberryPi. Depending on what is received from client, the pin 3 of the RaspberryPi (in my case is model B, rev 2) can be set to high or low. The server reasponds to client according to the action taken with the send_sck(clientsocket, msg).

Something like this:

while 1:
cmd = 0
mex = receive_sck(clientsocket) # lettura comandi veri
try:
mex = fine_data(mex)
print(&amp;amp;amp;quot;Incoming data from client: &amp;amp;amp;quot; + mex)
if mex=='on':
msg = 'Led ON.'
msg = msg_len(msg)
send_sck(clientsocket, msg)
cmd = 1
GPIO.output(3, 1)
if mex=='off':
msg = 'Led OFF.'
msg = msg_len(msg)
send_sck(clientsocket, msg)
cmd = 1
GPIO.output(3, 0)

if mex=='q':
#print(&amp;amp;amp;quot;Incoming data from client: &amp;amp;amp;quot; + mex)
msg = &amp;amp;amp;quot;Server ended. Stay rock.&amp;amp;amp;quot;
cmd = 1
msg = msg_len(msg)
send_sck(clientsocket, msg)
GPIO.output(3, 0)
break;
except:
print(&amp;amp;amp;quot;Uncorrect format received&amp;amp;amp;quot;)

if cmd == 0: # is still zero
msg = 'Unrecognized command.'
msg = msg_len(msg)
send_sck(clientsocket, msg)

server.close()
print(&amp;amp;amp;quot;Server connection closed.&amp;amp;amp;quot;)

Note that this GPIO library “simply” interacts with hardware control registers of the BCM2835 by accessing the file /dev/mem, and since it is a system file you must run the server script as a root. In the server script you should locate the line below and set the address and port like this

server.bind(('192.168.1.5', 8000))

while in the client

client.connect(('192.168.1.5', 8000))

The server should have the internal LAN address. The internal port, 8000 in this case, is the one used inside the LAN by the server. At the moment in which you decide to access from outside, you should open a port (can be different if you want) on the router allowing it to redirect the packets toward the internal address:port, here 192.168.1.5:8000. For this capability, refer to your specific router manual. About the port, an high number (> 1024) usually do not provides standard services, so often conflicts are not encoutered.

The client.connect() must use the address of your ISP provider if you are outside the LAN, and the proper port opened on your router previously. Here are the same because I’m using the scripts inside my LAN, so no redirect is needed. Note that you can’t test the correctness of forwarding from inside the same LAN where the server resides. If you are in the same LAN, you can instead connect to localhost and run both scripts on the same machine.
There are tons of guides on emulate a free static public IP and I don’t want to load too much this presentation. And since I’ve not tried this method in first person, I will skip the static IP tricks for now.

You can download the scripts and start playing with them. I’ve remained quite short in descriptions because I want to arrive at the next part. 🙂
If you have questions, you can ask them in the comments, of course!

And finally…

The final result is a LED which is on whenever I write “on” and viceversa if “off”. Connect using an SSH connection to the board and start the script “server.py” by typing

sudo python3 server.py

because I wrote it in Python3. If there is an error about an already used port:

[Errno : 98] Message Address already in use

it’s normal, maybe there is an open connection or if you are already ran the script, maybe the client has not closed the connection. In that case you can change the port: after few minutes that previous port will became again available.

I remotely driven the GPIO 3 (SCL) pin. Be careful to follow the guidelines  to avoid to ruin the SoC pins, and remember that those pins can output up to 50mA. See image below:

The setup is very trivial, just the GPIO 3 with a series resistor connected to the LED. This useless picture is posted “just for fun”. 🙂

ledThe server script has a console allowing to understand what is going on, while the client works as always in a console style. Whenever the LED is in the wanted state, the server confirm that by printing on screen. The use of the screen can be avoided if not needed, because the response message is sent to the client, as you can see.

serverThe client, for now, requires the user intervention, so the screen is needed to allow writing commands in console, as below.

client

 Downloads

Client:  https://www.dropbox.com/s/xzxoirvgg2hwm2u/client.py?dl=0

Server: https://www.dropbox.com/s/o6b0klsoqcymch8/server.py?dl=0

Network programming – an introduction

As a support towards a complete tutorial and presentation of the “An UART interpreter“, I discovered the need to learn something about networking under Python language. In this way, I should be capable to enable the access to Internet of my microcontroller based board.

So I will start by showing an example on how to send any signal to the other end of a network – here in the same localhost, i.e. send messages from my PC to itself, but once the mechanism is set, we can speak to the other side of the planet by simply changing the IP address. By doing this, the UART interpreter will be IoT-enabled.

For everyone who don’t know anything about networking softwares and modules and wants to learn something, like me, the best thing is to start from the bottom layer of the network: sockets. Which are not so bottom, but let accept this trade off. In these examples is developed a sample software in Python 3 running on two machines, here used the same PC speaking in local, i.e. the address 127.0.0.1, or localhost. I used Python because is well suited to easily work with the RaspberryPi, which will be exploited soon in next projects. And can be used to test the set of networking scripts described here.

What is a socket? Basically it’s an “istance” of two point of communication between processes, or in other words are an IPC methodology (Inter-Process Communication). You can use them on local machine or between machines separated by any kind of networks. Of course, there are more efficients methods to allow two processes to communicate on the same machine, but sockets are almost the only simple solution to allow communication over the network. From the user point of view, the socket represent an IP address to which you want to connect to by using a specific “port” that can (but don’t have to) be different depending whether it is a request or an actual data response. A port then can be considered as an instance of communication with that IP, which can be treated in a different way as different as the port chosen, for that reason you should not use ports with a fixed function and causing possible conflicts between processes. Use for example:

127.0.0.1:8000

Thanks to sockets, you can also see this website, where data pass through your IP at port 80. For a complete description I invite you to search around, since explaining them deeply is not the purpose of this article and I’m not capable to.

Server side

In Python, we need to import the module socket in order to use a server socket and binding it at localhost address. This is the server address, but can be anything else than localhost (127.0.0.1). Then we listen for a certain number of connections at a time, here just one connection at a time is allowed, but typically is 5. The socket is an INET type  since we deal with internet protocol addresses and is STREAM because I want to keep monitoring of the activity until I decide to disconnect from it. If this is DGRAM, I send my data and then disconnect. But here I want to disconnect at the end of the client execution only.

server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(('localhost', 8000))
server.listen(1)

Then in a loop we need to accept the connection, creating a client socket with its address, then receive from the client a string saved in mex and send the server aknowledge to the client.

while 1:
 #accept connections from outside
 print(&amp;quot;Server notification: waiting for client&amp;quot;)
 (clientsocket, address) = server.accept()
 mex = receive_sck(clientsocket)
 print(&amp;quot;Server notification: \n\tIncoming data: &amp;quot; + mex)

 msg = &amp;quot;Server notification: data received&amp;quot;
 msg = msg_len(msg)
 send_sck(clientsocket, msg)

A sad but true thing is that inside blocking sockets I can wait forever because there is not a prefixed size. So I’ve implemented by using the msg_len() method a protocol which attach before the message the number of chars followed by an underscore character. Of course, the number include the number of char used by the number it self, too. The method return the modified string.

In this way at receiving part I used the receive_sck()  to receive and check if in the received part the “_” is present, thanks to this I can identify the number. After that I set the MSGLEN to this number, exiting from the while after that amount of data is received, and if present, more data is discarded since is not part of the message, according to this protocol. Must be noticed that even if the maximum buffer allowed is 2048 chars, I can receive even only one char at a time, if the line is congested. So a marking char must be searched every time I receive from socket until I find it. The marking char here is “_”.

With the send_sck() is more simple, because what I send is known, but must be previously transformed to attach the number of chars, useful at the receiving part. When transmitting an error handling can be implemented.

Client side

On the other side of the connection the same methods are reused for modularity. They can be improved by using heavily a self created class, which I didn’t done here.

client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(('localhost', 8000))
msg = 'Hello IoT!'
msg = msg_len(msg)
send_sck(client, msg)
client.shutdown(1)
mex = receive_sck(client)
print(&amp;quot;Client notification: \n\tIncoming data: &amp;quot; + mex)
client.close()

Here I’ve implemented the socket client and connected it to server, the message msg is sent to the client socket called with fantasy client , sending the content to the server at localhost, i.e. the server address. The new thing is the client.shutdown(1), in which I close the client transmission capability allowing it to still receive, just to check the aknowledge from the server. Then I close the socket, disconnecting from server.

Conclusion

As a result, this is how two separate processes (server.py in the white backgroung and client.py executed on prompt of Windows) are communicatin each other. Client script is executed after server is waiting for client. Raw data received from each part is printed on stdout after the “Incoming data: ” and the actual payload contains the number of chars added by the protocol. These can be filtered out when printed, because are needed only by the algorithm and usually not by the user.

serverclient

Here you can dowload both client and server and play with them.

Source Code:
https://www.dropbox.com/s/766bokip1wogr0o/client.py?dl=0
https://www.dropbox.com/s/8fz2yz352r58ut1/server.py?dl=0

An UART interpreter – sneak peek

mainbrdIn a hot summer of a couple of years ago, trying to not thinking about the stifling heat of the afternoon, I put some efforts in a system which should drive a set of motors and peripherals by means of a single serial line. The idea consists in creating a sort of instruction set, sent over the serial, to transmit and receive informations, but the main effort was to develop this under a robust software, to make the system rensonsive and interrupt driven, fully aknowledged to keep trace of errors also.

This project is born following these ideas, on order of development:

  • Driving a low level peripheral using a console, with an affordable and error tolerant firmware
  • HW separated by SW, being able to design them finely and independently
  • Being able to drive the HW on any type of PCs
  • Expliting any software that can handle strings, the only specification is to know the language of the console
  • As a natural consequence, a PC connected on the Internet can drive the HW remotely

The main idea

In order to do this, I thought to a concept made of a two stages of control: the microcontroller receive commands from the user and put them in action. But the user of the microcontroller could be another board or PC, not necessarily a human generated commands. This, as previoulsy said, can be made through a console.

In this way the microcontroller carries an “interpreter” which translate commands to the hardware. By this mechanism I split a single action in a confined memory and execution time, affordable and in a responsive way. Due to this software “modularity” it is possible to extend indefinitely the instructions that the micro can interpret and use all of its calculus power to control precisely the hardware desired or manage the instructions in a robust way. The MCU is then only a front-end of the system which receive and sent signals from/to external world. By using a coherent language, the device can communicate with other peripherals (like a PC) by using the serial, and being driven by them directly.

The abstraction layer

The user only see a console over the serial, which work so vaguely similar to the Python line interpreter. But commands are simple, being able to be generated by a software running on another machine which send them to the console. And are simple because this project is at a very early stage. 🙂

Here the user of the console can implement all the algorithms, writing them down to the MCU generated command prompt. With this machanism we already have an abstraction to blink an LED over the Internet, or driving a robotic arm, nothing change inside the MCU. The abstraction give the possibility to personalize the behavior of the device and save in a text file on the PC without the need of reprogramming the chip. Also, but this is not structured yet, can be possible to save this content on an EEPROM or whatever memory support can be reached by the microcontroller, when the console is not directly accessible.

So, anyone can control the device directly or through a PC algorithm. The power of the idea is that if I want to drive a robotic arm, I have sufficient abstraction to drive it directly from another location, making this MCU implementation very suitable for remote control and environment analisys.

At this stage of the project, the software running on an internet connected device is written in the multi-platform Python language, due to its versatility. It reads the console and write commands to it. But here the console is just an echo taken from the microcontroller, elaborated by the Python based SW and printed on the screen. What is important, the user can not know anything of the hardware of the MCU, being free to think to the optimum driving algorithm.

Are also implemented 2 buttons where each control a device bypassing the physical serial console. But due to the modularity of the firmware, when a button is pressed, it is not activated a signal inside the MCU, but there is an auto-generated internal command sent back to the console, by parameter passing without using the USART module. By means of this mechanism, all controls are the same, the code remains modular and there are no performance losses.

A pretty stupid image is shown here, just to make the idea. The light green is what is done inside the MCU, while the light orange shows the basic algorithm of the PC. Between these two colors, there is an USART wrapper, but can be implemented on other technologies, like SPI, I2C, whatever isola supported on the MCU side.

prj

I almost forget an analogy: we can think about a client-server structure. Here, the server is the MCU while the client the Phython based PC software. I like thinking to the old fashioned gerarchy of the computer terminals, a dumb terminal that send commands on a mainframe. But here the terminal is not so dumb compared to those in 1960.

Some specifications

I almost forget to talk about the core. Its based on a 8-bit PIC18F47J53 microcontroller, which can handle  very efficiently two levels of interrupts priority. Here I can give more priority to the USART commands instead of those on board, or viceversa, depending on what I desire. For example, when I press a button and I want to be interrupted by the serial, I can put the serial receiving interrupt up in priority. The project was previously born on an Atmega328P by using Avrdude and AVR Studio 6, but then ported on this PIC using XC8 and MPLAB-X, due to better hardware specifications.

Moreover, and I will talk about that on next article, on the MCU board there are two buttons and here are configured to work efficiently like a joystick. The ADC samples a value of voltage on a potentiometer and can set the intensity of the command.

The software on PC is Python based and uses the Pyserial library.

epic

And now?

Now as soon as I can find some time to put something on more than a breadboard and when the Python based terminal is presentable (keep in mind what “presentable” means for an electronic student…), I will put something more that a bare useless presentation. I will also put the sources and I will explain better what is going on, trying to be helpful to who want to play with this stuff.

On the next article, I’ll try to explain how to control a motor by using an opposite phase PWM modulation, allowing the analog control of a motor. This can be done also without an MCU, by using only one 555. I’ll see what implementation I will adopt.

Nevertheless, this opposite phase PWM method can be integrated very well with this UART intepreter project: one can drive the motor manually and then an algorithm can decide if that speed must be kept, and so take the control over the device. Yes, I will decisely talk about that in the first release of this project. I will play with the 555 another time…
But I’m afraid that I’ll not find so much time to setup a proper sensor and data acquisition. So, I need to organize myself on order to find time, and this is not a good news.

Il tempo si paga in energia

723px-Neutrino_bubble_chamber_decay_overlay

Una frase del genere odora di Einstein. Ma lui rapportava la massa all’energia… ah, no, c’era di mezzo anche il tempo, con qualche giro matematico.
Passeggiando in giro, andando in bicicletta, muovendosi in acqua. Ci rendiamo tutti conto che quando le cose vengon fatte con calma paiono meno faticose. Basti pensare alla stanchezza dopo una lunga scalinata, fatta di corsa. E sì, uno può essere stanco anche dopo averla fatta lentamente, ma di certo è un qualcosa di più sopportabile. Questo nonostante l’energia fornita totale per salire gli scalini sia esattamente la stessa. E accettando l’ovvietà di questa conclusione con un po’ di umiltà, la mente si sposta di un passo avanti.

Ecco infatti che, come qualsiasi esperto può affermare, il nostro corpo – ma probabilmente tutte le forme di vita – può dare una certa quantità di energia in un lasso di tempo definito che è sempre inferiore alla potenziale quantità di energia che realmente possiede (riserve di grasso, ATP e tutto quel che ora si conosce a riguardo). Ed ecco che alla corsa verso la cima della scalinata, salvo essere allenati, le forze “mancano”. L’allenamento altro non è infatti che aumentare l’eneriga utilizzabile nello stesso lasso di tempo, o trovare l’equilibrio tra l’energia che si consuma e quella che si può fornire per migliorare la resistenza, sia che si parli di allenamento in palestra o allenamento mentale, nello studio. E quindi mi dico: ma solo io essere vivente? O anche ciò che mi circonda funziona così?

Allora penso alle forme inorganiche, come una goccia d’acqua che corrode una pietra. Dopo molto tempo, si osserverà lo stesso danno che avrebbe fatto un proiettile sparato sulla pietra, ma sarebbe avvenuto in una frazione di secondo. Il meccanismo di scambio di energia è completamente diverso dal corpo umano, ma in qualche modo c’è una filosofia di base che unisce cose e animali.

Poi però mentre scrivo, mi viene in mente che qualcuno, da poco usciti dal Medioevo, agli albori della matematica, ha tradotto questa “filosofia di base”. Si è arrivati ad una forma di equazione differenziale, perfettamente nota a chi ha studiato un minimo di fisica, la Potenza:

bfbd26cca59bdfa4546d8359b12c9760

Dove L è il lavoro, ovvero la quantità di energia spostata e t è il tempo.

Una cosa interessante è come tutte queste formule fanno fede ad un metodo inventato dalle persone (la Matematica), capace di prendere un fenomento e quantificarlo permettendo di gestirlo, in un linguaggio che non conosce nazioni. Ma se si riesce a concepire come innovativa una cosa così vecchia, allora scatta l’ultimo pensiero di questo articolo di dubbia utilità: le formule descrivono come le cose accadono, ma non il perché. O meglio, con esoterismi tipo fisica quantistica e compagnia bella, i perché di molte formule vengono svelati. Anzi, solo spostati un po’ più in la.

Dalla fatica che si può provare nel fare le cose in fretta, pare proprio che il tempo guadagnato (dipende…) si paghi in energia. Moneta di scambio per tutti gli esseri viventi che cercano solo un po’ di tempo in più. Se non esisti non hai energia, sei uno con cui alla banca del tempo non vogliono avere a che fare e resti lì, immemore e senza tempo.

Ma di questo ne parlerò un’altra volta, tempo e voglia permettendo. Volevo dire, energia da spostare permettendo.

The anti-loud siren

2a

What happens if some one try to piss off people and make them scream, and you can’t calm down them?
What happens if you are in a crowded environment where people start talking when they shouldn’t?
Or what happens if you are one of those people who talk constantly loud out, but wants to improve they ability to talk “normal”?
Or if your boss always scream at you?
WHAT if there is a solution to this?

Ok, after this junk styled presentation, I want to show a little project born and made in a week-end. So very stupid, but also nice, imho. It’s based on make some loud when there is too noise around and I decided to build it ‘cause sometimes people talks loudly and they didn’t notice that. Presented as a little joke for them, here is shown the anti-figher siren.

CONCEPT AND SCHEMATICS

Here we are talking about a simple analog circuit which receive a signal from a microphone, bias and amplifying it, then put into a common emitter BJT which square it up to be read from a microcontroller – here an Atmega328 MCU, Arduino Duemilanove board.
I started by thinking about what is already available at home. And I realized that I can finally use that lonely LM358 (dual op-amp), then an BC337 was also available and a mic was re-born from.. I don’t remember, it was simply there, but with no mark helping to understand its capabilities.
That being said, after few suggestion of order of amplification of the preamplifier, I came up with a solution which magnifiying the signal up to 2550 times approximately, reduced accordingly by the potentiometer. This includes the second amplifier which amplifies of 12 times and drive the bjt, common emitter to work in saturation. The biasing of the final signal allow the BJT to act as a switch “normally closed”: in this way the signal which allow the MCU pin change is related to the “negative” wave of sound signal, and the digital value is not inverted.

Here the schematic, in Eagle:

mic

Now, since this project was started in Friday morning, and I was pretending to finishing it by the weekend, the fastest and rough thing to do was to use Arduino and sketching a program that uses interrupt on the pin from the microphone and “tone-out” an alarm of 1 second on a buzzer. Actually, the alarm sounds like a siren, and to be honest I find it as cool as stupid.

Here the sketch:

 // Definition of interrupt names
#include < avr/io.h >
// ISR interrupt service routine
#include < avr/interrupt.h >

static uint8_t buzz = 0;
uint16_t i = 0;

ISR(PCINT2_vect) {
  buzz = 1;
}


void setup() {

  pinMode(4, INPUT);
  pinMode(5, OUTPUT);
  EIMSK = 0;
  PCICR = 0b00000100;
  PCMSK2 = 0b00010000; // pin4 digital arduino

  sei();
}

void loop() {

  if (buzz){
    for (i = 2000; i<3000; i=i+10){
       tone(5, i);
       delay(2);
       }
    noTone(5);
    for (i = 3000; i>2000; i=i-10){
       tone(5, i);
       delay(2);
    }
    noTone(5);
    buzz=0;
  }
}

FINAL WORK

The breadboarded version looks like this:

 bread

But the final work, if we can call it this way, is made on a prototyping board and overlapped to Arduino. Then fixed with some tape, can be hooked to an arm, like we can do with our smartphones when jogging, or can simply be put smewehere on the desk. This ugly superimposition of boards (and also glued with tape, but not applied when taking these pics), shows my main skill hadicap: enclosures. 🙂

Ah, components are not too close to board to allow me to eventually reuse them in the future, inside a breadboard. I know, I could avoid it, but it’s a trick in case of more expensive components which are mounted on temporary circuits without any socket – this is not the case, though.

 1

DSCN3693[1]

The perfect final of every project – no matter what how stupid it is, is to build a final prototype all in a single board or in a system which is made for this purpose and nothing more (if it is not desired in the project, obviously). So, if an MCU is used, a proper smaller one should be used. Or in this case, one can have some more fun by building a detector and siren generation circuits without the use of a microcontroller, and put everything toghether, obtaining a nicer gadget… To keep people silent!