How to generate intermediate files with clang/gcc to read Ruby's MRI implementation
I am currently diving into the MRI implementation of Ruby. It is really interesting to see how your beloved language concepts are actually implemented in plain C. The MRI code is actually pretty easy to read and you can read the implementations for Array or Hash to get an idea.
If you already read a bit C code, you probably noticed tons of #define
directives preferably at the beginning of header files. These configure mostly
platform specific things. I came to this point when I digged through the array
implemention and found this line:
# array.c https://github.com/ruby/ruby/blob/439224a5904411b288e441096e21a41244ddd1d6/array.c#L25
VALUE rb_cArray;
This shows a custom type definition in C. To know how VALUE is defined, I had to read the ruby.h file:
# include/ruby/ruby.h https://github.com/ruby/ruby/blob/439224a5904411b288e441096e21a41244ddd1d6/include/ruby/ruby.h#L103-L124
#if defined HAVE_UINTPTR_T && 0
typedef uintptr_t VALUE;
typedef uintptr_t ID;
# define SIGNED_VALUE intptr_t
# define SIZEOF_VALUE SIZEOF_UINTPTR_T
# undef PRI_VALUE_PREFIX
#elif SIZEOF_LONG == SIZEOF_VOIDP
typedef unsigned long VALUE;
typedef unsigned long ID;
# define SIGNED_VALUE long
# define SIZEOF_VALUE SIZEOF_LONG
# define PRI_VALUE_PREFIX "l"
#elif SIZEOF_LONG_LONG == SIZEOF_VOIDP
typedef unsigned LONG_LONG VALUE;
typedef unsigned LONG_LONG ID;
# define SIGNED_VALUE LONG_LONG
# define LONG_LONG_VALUE 1
# define SIZEOF_VALUE SIZEOF_LONG_LONG
# define PRI_VALUE_PREFIX PRI_LL_PREFIX
#else
# error ---->> ruby requires sizeof(void*) == sizeof(long) or sizeof(LONG_LONG) to be compiled. <<----
#endif
VALUE could be one of uintptr_t, unsigned long or unsigned LONG_LONG. To find
out what it actually is, I could dig deeper in the code and see how all the
ifs resolve on my platform. But to make it easier for myself I can also use the
compiler to create so called intermediate files which already set all the defines.
I managed to generate a ruby.i file on OSX with the following command:
clang -I .ext/include/x86_64-darwin15/ -I include/ -x c -S -save-temps include/ruby/ruby.h
The -x c was necessary for me (probably not with gcc), otherwise clang fails with the following error:
error: invalid value 'c-header-cpp-output' in '-x c-header-cpp-output'
After adding this option, clang generated the i-file which contains the actual defines:
# 110 "include/ruby/ruby.h"
typedef unsigned long VALUE;
The full ruby.i file is in this gist.
Yay! Now I can continue reading the source code of MRI.
Note: Since this is a general feature of the compiler, you can do it with every C code you want to understand better.
Happy reading!
Logstash input plugin exec examples
Logstash is an awesome tool to grab not only lines from log files but also from tons of different sources. A very useful input plugin is exec. It can execute scripts or any other binary directly and capture the output to move it directly into Elasticsearch. Here are two examples where I am parsing arbitrary data, structurize it and put it in a remote Elasticsearch for further exploration with Kibana.
input {
exec {
command => '/home/pi/smaread'
interval => 300
type => 'solar'
}
}
filter {
if [message] != "" {
csv {
separator => ';'
columns => ['Iac', 'Uac', 'Fac', 'Pac', 'Zac', 'Riso', 'dI', 'Upv-Ist', 'PPV', 'E-Total', 'h-Total', 'h-On', 'Netz-Ein', 'E-Total DC', 'unknown', 'Status', 'Fehler']
remove_field => ['column18']
}
mutate {
rename => { "Iac" => "iac" }
rename => { "Uac" => "uac" }
rename => { "Fac" => "fac" }
rename => { "Pac" => "pac" }
rename => { "Zac" => "zac" }
rename => { "Riso" => "riso" }
rename => { "dI" => "di" }
rename => { "Upv-Ist" => "upv_ist" }
rename => { "PPV" => "ppv" }
rename => { "E-Total" => "e_total" }
rename => { "h-Total" => "h_total" }
rename => { "h-On" => "h_on" }
rename => { "Netz-Ein" => "netz_ein" }
rename => { "E-Total DC" => "e_total_dc" }
rename => { "Status" => "status" }
rename => { "Fehler" => "fehler" }
convert => {
'iac' => 'float'
'uac' => 'float'
'fac' => 'float'
'pac' => 'float'
'zac' => 'float'
'riso' => 'float'
'di' => 'float'
'upv_ist' => 'float'
'ppv' => 'float'
'e_total' => 'float'
'h_total' => 'float'
'h_on' => 'float'
'netz_ein' => 'float'
'e_total_dc' => 'float'
'unknown' => 'float'
'status' => 'float'
'fehler' => 'float'
}
}
}
}
output {
#test it with
#stdout { codec => rubydebug }
if [type] == 'solar' and [message] != "" {
elasticsearch {
hosts => ['es.yourdomain.com']
user => 'user'
password => 'password'
index => "solar"
}
}
}
input {
exec {
command => '/home/pi/vpro -2 -C tipmod -x -d 20 /dev/ttyWetter'
interval => 300
type => 'weather'
}
}
filter {
kv {
field_split => "\n"
value_split => " = "
}
mutate {
rename => { "rtBaroTrend" => "rtbarotrend" }
rename => { "rtBaroTrendImg" => "rtbarotrendimg" }
rename => { "rtBaroCurr" => "rtbarocurr" }
rename => { "rtInsideTemp" => "rtinsidetemp" }
rename => { "rtInsideHum" => "rtinsidehum" }
rename => { "rtOutsideTemp" => "rtoutsidetemp" }
rename => { "rtOutsideHum" => "rtoutsidehum" }
rename => { "rtWindSpeed" => "rtwindspeed" }
rename => { "rtWindAvgSpeed" => "rtwindavgspeed" }
rename => { "rtWindDir" => "rtwinddir" }
rename => { "rtWindDirRose" => "rtwinddirrose" }
rename => { "rtWindChill" => "rtwindchill" }
rename => { "rtHeatIndex" => "rtheatindex" }
rename => { "rtRainRate" => "rtrainrate" }
rename => { "rtIsRaining" => "rtisraining" }
rename => { "rtUVIndex" => "rtuvindex" }
rename => { "rtSolarRad" => "rtsolarrad" }
rename => { "rtRainStorm" => "rtrainstorm" }
rename => { "rtStormStartDate" => "rtstormstartdate" }
rename => { "rtDayRain" => "rtdayrain" }
rename => { "rtMonthRain" => "rtmonthrain" }
rename => { "rtYearRain" => "rtyearrain" }
rename => { "rtDayET" => "rtdayet" }
rename => { "rtDayETmL" => "rtdayetml" }
rename => { "rtMonthET" => "rtmonthet" }
rename => { "rtYearET" => "rtyearet" }
rename => { "rtXmitBattt" => "rtxmitbattt" }
rename => { "rtBattVoltage" => "rtbattvoltage" }
rename => { "rtForeIcon" => "rtforeicon" }
rename => { "rtForeRule" => "rtforerule" }
rename => { "rtForecast" => "rtforecast" }
rename => { "rtSunrise" => "rtsunrise" }
rename => { "rtSunset" => "rtsunset" }
rename => { "rtCaptureTimestamp" => "rtcapturetimestamp" }
convert => {
"rtbarocurr" => "float"
"rtinsidetemp" => "float"
"rtoutsidetemp" => "float"
"rtwindchill" => "float"
"rtheatindex" => "float"
"rtrainrate" => "float"
"rtrainstorm" => "float"
"rtdayrain" => "float"
"rtmonthrain" => "float"
"rtyearrain" => "float"
"rtdayet" => "float"
"rtdayetml" => "float"
"rtmonthet" => "float"
"rtyearet" => "float"
"rtbattvoltage" => "float"
"rtinsidehum" => "integer"
"rtoutsidehum" => "integer"
"rtwindspeed" => "integer"
"rtwindavgspeed" => "integer"
"rtwinddir" => "integer"
"rtxmitbattt" => "integer"
"rtforeicon" => "integer"
"rtforerule" => "integer"
}
}
}
output {
#test it with
#stdout { codec => rubydebug }
if [type] == "weather" {
elasticsearch {
hosts => ['es.yourdomain.com']
user => 'user'
password => 'password'
index => "weather"
}
}
}