Changes

Page history
more wiki authored by mitshell's avatar mitshell
Show whitespace changes
Inline Side-by-side
Compiling-asn1-specifications.md
View page @ 15b230c6
# How to compile ASN.1 specifications with the pycrate ASN.1 compiler
TODO
## What is ASN.1
ASN.1 is a notation used to describe abstract structures.
Those are made from basic types:
- NULL
- BOOLEAN
- INTEGER
- REAL
- ENUMERATED
- BIT STRING
- OCTET STRING
- *String (many String types are defined, each using a different codec)
- UTCTime and GeneralizedTime
from constructed types:
- CHOICE
- SEQUENCE and SET
- SEQUENCE OF and SET OF
and from ASN.1 specific types:
- OBJECT IDENTIFIER and RELATIVE-OID
- EXTERNAL, EMBEDDED PDV and CHARACTER STRING
After structures are defined according to these types, encoders must be used
to serialize data. This enables heterogeous systems to communicate in an unified
way according to a given ASN.1 definition.
The different encoders defined are:
- BER (Basic Encoding Rules), CER (Canonical Encoding Rules) and DER (Distinguished
Encoding Rules): byte-oriented codecs
- PER (Packed Encoding Rules), aligned and unaligned: bit-oriented codecs
- XER (XML Encoding Rules) and GSER (Generic String Encoding Rules): text-oriented
The current specification for ASN.1 and its codecs can be found in ITU-T documents:
- [X.680](http://itu.int/ITU-T/X.680), for the ASN.1 notation syntax
- [X.681](http://itu.int/ITU-T/X.681), for specific CLASS objects
- [X.682](http://itu.int/ITU-T/X.682), for specific constraints
- [X.683](http://itu.int/ITU-T/X.683), for parameterization
- [X.690](http://itu.int/ITU-T/X.690), for BER, CER and DER codecs
- [X.691](http://itu.int/ITU-T/X.691), for PER codecs
- [X.693](http://itu.int/ITU-T/X.693), for the XER codec
- GSER codec is defined by IETF under the [RFC 3641](https://tools.ietf.org/html/rfc3641).
ASN.1 was originally specified in 1988 in the ITU-T recommendation X.208.
Good ressources about ASN.1 can be found in those two books that are available for free
on the web:
- *ASN.1 - Communication Between Heterogeneous Systems*, by Olivier Dubuisson
and translated by Philippe Fouquart
- *ASN.1 Complete*, by John Larmouth
Here are some other available useful ressources:
- OSS Nokalva provides an online ASN.1 compiler, plus multiple encoders and
decoders at the [asn1-playground](http://asn1-playground.oss.com).
- Lev Walkin proposes [asn1c](http://lionet.info/asn1c/), a very complete ASN.1
to C and C++ compiler plus multiple encoders and decoders (and also an online
one).
- Fabrice Bellard proposes [ffasn1](http://bellard.org/ffasn1), which has a free
ASN.1 message converter and editor which supports many encoding rules too.
- ITU-T has a complete web page referencing tools and softwares supporting ASN.1
at the [ITU-T ASN.1 tools](http://www.itu.int/en/ITU-T/asn1/Pages/Tools.aspx).
## Support in pycrate
The current support of the ASN.1 syntax in the pycrate ASN.1 compiler corresponds
to the last ASN.1 versions, i.e. the 2015 one, which is itself quite similar to
the ASN.1 standard from 2002.
There are still few limitations in the current version of the compiler (see
[limitations](#limitations)).
In particular, the following general features are supported:
- automatic and manual tagging
- type extensibility
- constraints: value and range, size, permitted alphabet
- CLASS objects (also called information objects), open types and table constraints
- parameterization
- ANY type and the *DEFINED BY* construction (required for some older specifications)
The pycrate ASN.1 runtime supports the following encoders:
- BER, CER and DER
- PER aligned and unaligned
- ASN.1 textual value notation
and the following features:
- table constraint resolution at runtime, hence resolving open types when possible
- fragmentation in BER, CER and PER (DER does not use fragmentation)
- runtime checking against value, range and size constraints (which can be disabled)
## Limitations
There are still few limitations in the current code.
On the compiler side:
- the left part of an ASN.1 assignment and the assignment sign (*::=*) have to be on a single line
- C-style comments between `/*` and `*/` are not supported
- old-school ASN.1 MACROs are not supported, therefore the compiler cannot process SNMP MIBs
- some extreme parameterization cases can lead to a compiler error (see the object
*AllPackagesAS* in the ITU-T Q.775 specification)
- recursive or circular objects definition together with parameterization are not supported
- some constraints which do not have any impact on encoding rules are barely or not processed,
this includes WITH COMPONENT (WITH COMPONENTS is however processed), PATTERN, SETTINGS and
CONSTRAINED BY constraints
- the constraint ENCODE BY is not processed either
- new universal objects are not supported, this includes DATE, TIME-OF-DAY, DATE-TIME, DURATION,
OID-IRI and RELATIVE-OID-IRI objects
All those cases are rarely found into classical ASN.1 specifications, and would require
some more code to be supported. This is mostly why they are not supported, yet.
On the runtime side:
- XER and any XML-related features are not supported
- OER, GSER and other encoding rules are not supported
- ECN is not supported either
- some codecs for specific *String* types are not supported (mainly because Python does not
have support for them): TeletexString, VideotextString, GraphicString and GenericString,
all using the ISO-2022 encoding.
## How to
In order to work with an ASN.1-specified protocol or data format, here are the two steps
to follow:
- compile the ASN.1 definition into Python source code
- use the generated Python code to encode specific values that you assign to given
objects, and decode bytes' buffers to specific values
In order to compile an ASN.1 specification, few possibilities are offered.
### the tool *pycrate_asn1compile.py*
A specific tool
[tools/pycrate_asn1compile.py](https://github.com/ANSSI-FR/pycrate/blob/master/tools/pycrate_asn1compile.py)
is provided. Just requests its help to see how it works:
$ ./tools/pycrate_asn1compile.py --help
usage: pycrate_asn1compile.py [-h] [-i INPUT [INPUT ...]] [-o OUTPUT] [-j]
[-fautotags] [-fextimpl] [-fverifwarn]
compile ASN.1 input file(s) for the pycrate ASN.1 runtime
optional arguments:
-h, --help show this help message and exit
-i INPUT [INPUT ...] ASN.1 input file(s) or directory
-o OUTPUT compiled output Python (and json) source file(s)
-j output a json file with information on ASN.1 objects
dependency
-fautotags force AUTOMATIC TAGS for all ASN.1 modules
-fextimpl force EXTENSIBILITY IMPLIED for all ASN.1 modules
-fverifwarn force warning instead of raising during the
verification stage
It takes a directory containing ASN.1 definitions (file with *.asn[1]* suffix),
or one or multiple files as input, and generates a Python source code as output.
It can also generate a json file detailing ASN.1 objects' dependency, with the
*-j* option. Few more options starting with *-f*' are available to be passed to
the compiler, in order to force specific behaviours.
Here is an example with the X2AP protocol ASN.1 definition:
$ ./tools/pycrate_asn1compile.py -i ./pycrate_asn1dir/3GPP_EUTRAN_X2AP_36423/ -o x2ap -j
[proc] module X2AP-Containers (oid: [0, 4, 0, 0, 21, 3, 2, 1, 5]): 16 ASN.1 assignments found
[proc] module X2AP-CommonDataTypes (oid: [0, 4, 0, 0, 21, 3, 2, 1, 3]): 10 ASN.1 assignments found
[proc] module X2AP-IEs (oid: [0, 4, 0, 0, 21, 3, 2, 1, 2]): 342 ASN.1 assignments found
[proc] module X2AP-PDU-Contents (oid: [0, 4, 0, 0, 21, 3, 2, 1, 1]): 250 ASN.1 assignments found
[proc] module X2AP-Constants (oid: [0, 4, 0, 0, 21, 3, 2, 1, 4]): 233 ASN.1 assignments found
[proc] module X2AP-PDU-Descriptions (oid: [0, 4, 0, 0, 21, 3, 2, 1, 0]): 36 ASN.1 assignments found
--- compilation cycle ---
--- compilation cycle ---
--- compilation cycle ---
--- compilation cycle ---
--- verifications ---
[proc] ASN.1 modules processed: ['X2AP-Containers', 'X2AP-CommonDataTypes', 'X2AP-IEs', 'X2AP-PDU-Contents', 'X2AP-Constants', 'X2AP-PDU-Descriptions']
[proc] ASN.1 objects compiled: 421 types, 198 sets, 262 values
[proc] done
$ ls -l x2ap*
-rw-rw-r-- 1 user user 255442 Oct 10 14:07 x2ap.json
-rw-rw-r-- 1 user user 1452734 Oct 10 14:07 x2ap.py
From here, the *x2ap.py* and *x2ap.json* can be used as explained in the wiki part
on the [ASN.1 runtime](./Using-the-pycrate-asn1-runtime.md):
>>> import x2ap
>>> # do whatever encodings / decodings you want with this generated module
### the *pycrate_asn1c* modules
The ASN.1 compiler actually lies in the file
[pycrate_asn1c/proc.py](https://github.com/ANSSI-FR/pycrate/blob/master/pycrate_asn1c/proc.py).
The function *compile_text()* takes a textual input containing ASN.1 module(s), or
an iterable of textual inputs, and process them to build all the ASN.1 modules into a
dictionnary in the global class called *GLOBAL*.
The compiler function can take specific keywords to force specific behaviours if required:
- *autotags=True*
- *extimpl=True*
- *verifwarn=True*
Those corresponds to the behaviours described in the help string returned by the
*pycrate_asn1compile.py* tool.
Here is an example with the ASN.1 test file provided in the test directory:
>>> from pycrate_asn1c.proc import *
>>> help(compile_text)
[...]
>>> asntxt = open('./test/res/Hardcore.asn').read()
>>> compile_text(asntxt)
[proc] module HardcoreSyntax (oid: []): 116 ASN.1 assignments found
--- compilation cycle ---
--- compilation cycle ---
--- compilation cycle ---
--- verifications ---
[proc] ASN.1 modules processed: ['HardcoreSyntax']
[proc] ASN.1 objects compiled: 75 types, 3 sets, 37 values
[proc] done
>>> list(GLOBAL.MOD.keys())
['_IMPL_', '_USER_', 'HardcoreSyntax']
>>> GLOBAL.MOD['HardcoreSyntax']
{
_name_: 'HardcoreSyntax',
_oidstr_: '',
_oid_: [],
_tag_: 'AUTOMATIC',
_ext_: None,
_exp_: None,
[...]
Recur1: <Recur1 (SEQUENCE)>,
Recur2: <Recur2 (SEQUENCE)>,
ASNWrapper: <ASNWrapper (SEQUENCE)>
}
In order to process a single ASN.1 definition corresponding to an ASN.1 object,
most of the ASN.1 to Python translation is done by the *ASN1Obj* class in the file
[pycrate_asn1c/asnobj.py](https://github.com/ANSSI-FR/pycrate/blob/master/pycrate_asn1c/asnobj.py).
This class has actually many methods, most of them being called to parse specific
ASN.1 expressions and syntaxes.
After the *GLOBAL* class has been populated with Python objects built from the
ASN.1 definition module(s) provided as input, the code generator must be called
to generate a usable source file.
Two different code generators are currently provided:
- *PycrateGenerator*: generates one Python source file for the pycrate ASN.1 runtime
- *JSONDepGraphGenerator*: generates one json file describing objects' dependencies
The function *generate_modules()* must be called with the code generator as 1st
argument, and the destination file as 2nd argument. This will produce a source
file at the given destination:
>>> generate_modules(PycrateGenerator, '/tmp/hardcore.py')
>>> generate_modules(JSONDepGraphGenerator, '/tmp/hardcore.json')
### the *pycrate_asn1dir* directory
Many ready-to-use ASN.1 modules are provided in the
[pycrate_asn1dir/](https://github.com/ANSSI-FR/pycrate/blob/master/pycrate_asn1dir/)
directory. All ASN.1 definitions are placed in subdirectories, and corresponding
Python and json source files are available directly there.
The links between those subdirectories and Python and json files are set
in [pycrate_asn1c/specdir.py](https://github.com/ANSSI-FR/pycrate/blob/master/pycrate_asn1c/specdir.py).
It is possible to recompile a single ASN.1 specification, some of them, or
all of them, with the *generate_all()* function in
[pycrate_asn1c/proc.py](https://github.com/ANSSI-FR/pycrate/blob/master/pycrate_asn1c/proc.py).
This function takes a dictionnary as argument, linking destination name to subdirectory name.
For instance, `{'X2AP': '3GPP_EUTRAN_X2AP_36423', ...}`.
The default argument is the dictionnary *ASN_SPECS* from the *specdir.py* file.
So, it is possible to add a subdirectory in this *pycrate_asn1dir* directory,
and a corresponding entry in the *ASN_SPECS* dictionnary. This will integrate the subdirectory
as part of the default ASN.1 modules in the pycrate library.
Finally, it is possible to launch a recompilation of all ASN.1 modules, as set in the
*ASN_SPECS* file by simply executing the *proc.py* Python file.
This will call *generate_all(ASN_SPECS)* and regenerate all *.py* and *.json* files
in *pycrate_asn1dir*:
$ python -m pycrate_asn1c.proc
[GEN] LPPa
[proc] starting with ASN.1 specification: 3GPP_EUTRAN_LPPa_36455
[proc] module LPPA-CommonDataTypes (oid: [0, 4, 0, 0, 21, 3, 6, 1, 3]): 11 ASN.1 assignments found
[proc] module LPPA-Constants (oid: [0, 4, 0, 0, 21, 3, 6, 1, 4]): 36 ASN.1 assignments found
[proc] module LPPA-Containers (oid: [0, 4, 0, 0, 21, 3, 6, 1, 5]): 16 ASN.1 assignments found
[proc] module LPPA-IEs (oid: [0, 4, 0, 0, 21, 3, 6, 1, 2]): 75 ASN.1 assignments found
[proc] module LPPA-PDU-Contents (oid: [0, 4, 0, 0, 21, 3, 6, 1, 1]): 35 ASN.1 assignments found
[proc] module LPPA-PDU-Descriptions (oid: [0, 4, 0, 0, 21, 3, 6, 1, 0]): 18 ASN.1 assignments found
--- compilation cycle ---
--- verifications ---
[proc] ASN.1 modules processed: [u'LPPA-CommonDataTypes', u'LPPA-Constants', u'LPPA-Containers', u'LPPA-IEs', u'LPPA-PDU-Contents', u'LPPA-PDU-Descriptions']
[proc] ASN.1 objects compiled: 106 types, 32 sets, 47 values
[proc] done
[GEN] TAP3
[proc] starting with ASN.1 specification: GSMA_TAP3_17102014
[proc] module TAP-0312 (oid: []): 329 ASN.1 assignments found
[proc] module RAP-0105 (oid: []): 44 ASN.1 assignments found
--- compilation cycle ---
--- verifications ---
[proc] ASN.1 modules processed: [u'TAP-0312', u'RAP-0105']
[proc] ASN.1 objects compiled: 371 types, 0 sets, 0 values
[proc] done
[...]
## Working on compiled ASN.1 modules
It can be interesting to work on Python objects built in the *GLOBAL.MOD* dict
just after a compilation stage, instead of working with the Python module generated.
Those Python objects contain extended information, compared to the objects in
the Python module generated:
- all ASN.1 objects compiled have a *_ref* attribute which lists all other ASN.1 objects referenced
by them
- set of values are organized in *root* and *ext* lists, instead of specific ASN1Set objects
- parameterized objects are defined with specific references to their formal parameters;
moreover, they have a *_param* attribute which lists their parameters and corresponding
pathes into themselves
Using the *HardcoreSyntax.asn* module already compiled, we can illustrate this:
>>> Mod = GLOBAL.MOD['HardcoreSyntax']
>>> Mod['Seq2A']._ref # listing all references to other ASN.1 objects
{ASN1RefSet(HardcoreSyntax.Test3),
ASN1RefClassField(HardcoreSyntax.TEST3.&bool),
ASN1RefClassField(HardcoreSyntax.TEST3.&index),
ASN1RefClassField(HardcoreSyntax.TEST3.&Type)}
>>> Mod['INT02A']._param # listing formal parameters and corresponding pathes
{
tag: {'ref': [['tag', 0]], 'gov': <tag (INTEGER): >},
st: {'ref': [['cont', 'first']], 'gov': <st (INTEGER): >},
nd: {'ref': [['cont', 'second']], 'gov': <nd (INTEGER): >},
lo: {'ref': [['const', 0, 'root', 0, 'lb']], 'gov': <lo (INTEGER): >},
hi: {'ref': [['const', 0, 'root', 0, 'ub']], 'gov': <hi (INTEGER): >}
}
>>> Mod['Test3']._val # listing root and extended parts of a set of values
{'ext': None,
'root': [{
index: 1,
bool: True,
Type: <Type ([Real000] REAL)>
},
[...]
{
index: 6,
bool: False,
Type: <Type ([Seq000] SEQUENCE)>
}]}
Last but not least (but not very usable with real-world specification, too :),
the json file generated by the compiler can be visualized in the browser thanks to this
[D3-based javascript](https://bl.ocks.org/mbostock/4062045). So you can (try to) visualize
all the dependencies between ASN.1 objects from a compiled specification !