From 03dd4b92be7dcd5c8ab06da3810887060e06abd5 Mon Sep 17 00:00:00 2001 From: Igor Skochinsky Date: Mon, 3 Apr 2017 15:13:46 +0200 Subject: nwztools/database: misc improvements * make gen_db.py work on Windows/Python 2 - use hashlib module instead of md5sum, also don't rely on / for file path matching - don't use 'file' for a variable name * fix parse_nvp_header.sh for older kernels pre-emmc kernel sources use a slightly different #define format; adjust regexp to catch it. * add nwz-x1000 series NVP layout (from icx1087_nvp.h) some new tags have no description, alas the driver doesn't have them :/ * minor fixes to nvp/README fixed typos/wording Change-Id: I77d8c2704be2f2316e32aadcfd362df7102360d4 --- utils/nwztools/database/nvp/README | 39 ++++++------ utils/nwztools/database/nvp/nwz-x1000.txt | 82 +++++++++++++++++++++++++ utils/nwztools/database/nvp/parse_nvp_header.sh | 7 ++- utils/nwztools/database/nvp/parse_nvp_nodes.sh | 5 +- 4 files changed, 111 insertions(+), 22 deletions(-) create mode 100644 utils/nwztools/database/nvp/nwz-x1000.txt (limited to 'utils/nwztools/database/nvp') diff --git a/utils/nwztools/database/nvp/README b/utils/nwztools/database/nvp/README index cb403291c8..5ff18513d7 100644 --- a/utils/nwztools/database/nvp/README +++ b/utils/nwztools/database/nvp/README @@ -1,38 +1,39 @@ The NVP map varies a lot from players to players, it is inconceivable to build it by hand. The approach taken is to extract it from the kernel of each player. Since Sony provides the kernel of all players, it is 'only' a matter of -downloading all of them. A bit of back story on the NVP: it is non-volatile -area of the flash that is divided in regions and then "sectors" (unrelated to -hard drive sectors). Each "sector" stores the data of a "node". The ABI +downloading all of them. A bit of background on the NVP: it is non-volatile +partition of the flash that is divided in regions and then "zones". +Each "zone" stores the data of a "node". The ABI between the NVP driver and the userspace is an index: the userspace gives the -index of a node, and then drives looksup its table to see where it is and what +index of a node, and then drives looks up its table to see where it is and what is its size. The index map changes over time so Sony introduces standard "names" for its entries, those are 3-letters acronym (for example "fup" or "bti" or "shp") -that have a corresponding index. For some reason, the driver also contains a -description of the nodes, in english (so "bti" stands for "boot image"). +that have a corresponding index. Sometimes the driver also contains a +description of the nodes, in English (e.g. "bti" stands for "boot image"). parse_nvp_header.sh =================== -This script is given a file name, a kernel directory or a kernel tgz and it will +This script takes a header filename, a kernel directory or a kernel tgz and will try to extract the mapping automatically. It produces a list of pairs , where is the index of the node (that's the only thing that is usable on -a running device) and is the standard name of the node. Note that is -some kind of acronym (like 'fup') and the description needs to be generated -separatly (see other section). +a running device) and is the standard name of the node. The output should +be written to .txt +Note that is an acronym (like 'fup') and the description needs to be generated +separately (see nvprool section). parse_all_nvp_headers.sh ======================== -This scripts expects a directory to have the following structure: +This script expects a directory to have the following structure: dir/ nwz-a10/ linux-kernel-*.tgz nwz-e460/ - linxu-kernel-*.tgz + linux-kernel-*.tgz ... -Each sudirectory must the series name (as used in ../series.txt) and the kernel +Each sudirectory must be the series name (as used in ../series.txt) and the kernel must be a tgz (end in .tgz and not .tar.gz) of the form linux-kernel-*.tgz. Usually the variable bit will be the version but some kernels have unknown versions. It will then run parse_nvp_header.sh on each of them and store the result in a @@ -43,24 +44,24 @@ NOTE: the kernel can be symlinks to other files nvptool ======= -The kernel headers do no contain the description of the nvp node names. -This one can be extract from the icx_nvp[_emmc].ko driver on target using complicated -elf parsing done by nvptool. Technically nvptoo can find much more information -like the node -> humanname mapping as well and the actual sector on the disk but +The kernel headers do not contain descriptions of the nvp node names. +They can be extracted from the icx_nvp[_emmc].ko driver on target using complicated +elf parsing done by nvptool. Technically nvptool can discover much more information +like the node -> human name mapping as well and the actual zone in the flash but since we can already extract it easily from the headers, we only extract description names from it. parse_all_nvp_nodes.sh ====================== -This scripts expects a directory to have the following structure: +This script expects a directory to have the following structure: dir/ nwz-a10/ rootfs.tgz nwz-e460/ rootfs.tgz ... -Each sudirectory must the series name (as used in ../series.txt) and the rootfs +Each sudirectory must be the series name (as used in ../series.txt) and the rootfs must be a tar. It will then extract the relevant icx_nvp driver from it and run nvptool on it to produce a file called nodes-.txt diff --git a/utils/nwztools/database/nvp/nwz-x1000.txt b/utils/nwztools/database/nvp/nwz-x1000.txt new file mode 100644 index 0000000000..ab0f7fb872 --- /dev/null +++ b/utils/nwztools/database/nvp/nwz-x1000.txt @@ -0,0 +1,82 @@ +app,0 +bti,1 +hdi,2 +cng,3 +ser,4 +dg0,5 +dg1,6 +dcc,7 +mdl,8 +fup,9 +bok,10 +shp,11 +dba,12 +dbv,13 +tr0,14 +tr1,15 +mid,16 +tst,17 +gty,18 +fui,19 +lbi,20 +dor,21 +edw,22 +ubp,23 +syi,24 +var,25 +pcd,26 +dbs,27 +rnd,28 +ufn,29 +sdp,30 +ncp,31 +kas,32 +pnc,33 +rtc,34 +bpr,35 +e00,36 +e01,37 +e02,38 +e03,39 +e04,40 +e05,41 +e06,42 +e07,43 +e08,44 +e09,45 +e10,46 +e11,47 +e12,48 +e13,49 +e14,50 +e15,51 +e16,52 +e17,53 +e18,54 +e19,55 +e20,56 +e21,57 +e22,58 +e23,59 +e24,60 +e25,61 +e26,62 +e27,63 +e28,64 +e29,65 +e30,66 +e31,67 +clv,68 +slp,69 +ipt,70 +ep0,71 +ep1,72 +ep2,73 +ep3,74 +pts,75 +skt,76 +mac,77 +apd,78 +blf,79 +hld,80 +skd,81 diff --git a/utils/nwztools/database/nvp/parse_nvp_header.sh b/utils/nwztools/database/nvp/parse_nvp_header.sh index 8baab3c9eb..ee2be93516 100755 --- a/utils/nwztools/database/nvp/parse_nvp_header.sh +++ b/utils/nwztools/database/nvp/parse_nvp_header.sh @@ -66,13 +66,16 @@ else >&2 echo "Analyzing $FILE" fi +# old format: #define ICX1087_NVP_NODE_APP "/dev/icx1087_nvp/0" +# new format: #define ICX_NVP_NODE_APP ICX_NVP_NODE_BASE "0" + cat "$FILE" | awk ' \ BEGIN { \ - expr = "#define[[:space:]]+ICX_NVP_NODE_([[:alnum:]]+)[[:space:]]+ICX_NVP_NODE_BASE[[:space:]]*\"([[:digit:]]+)\""; \ + expr = "#define[[:space:]]+ICX[[:digit:]]*_NVP_NODE_([[:alnum:]]+)[[:space:]]+(ICX_NVP_NODE_BASE[[:space:]]*\"|\"/dev.*_nvp/)([[:digit:]]+)\""; } \ { \ if($0 ~ expr) \ { \ - print(tolower(gensub(expr, "\\1,\\2", "g", $0))); + print(tolower(gensub(expr, "\\1,\\3", "g", $0))); } \ }' diff --git a/utils/nwztools/database/nvp/parse_nvp_nodes.sh b/utils/nwztools/database/nvp/parse_nvp_nodes.sh index 456a707e5d..8fc4367a8f 100755 --- a/utils/nwztools/database/nvp/parse_nvp_nodes.sh +++ b/utils/nwztools/database/nvp/parse_nvp_nodes.sh @@ -4,6 +4,9 @@ # parse_nodes.sh /path/to/rootfs/dir output_file # parse_nodes.sh /path/to/rootfs.tgz output_file # +if [ -z "$NVP_LOG" ]; then + NVP_LOG=/dev/null +fi if [ "$#" -lt 2 ]; then >&2 echo "usage: parse_header.sh /path/to/icx_nvp.ko|/path/to/rootfs/dir|/path/to/rootfs.tgz output_file" exit 1 @@ -56,4 +59,4 @@ else >&2 echo "Analyzing $FILE" fi -./nvptool -x "$FILE" -o "$OUTPUT" >/dev/null +./nvptool -x "$FILE" -o "$OUTPUT" >"$NVP_LOG" -- cgit