The XML data file and scripts uploaded today are the simple beginnings of a framework to allow access to folders containing USFM texts. USFMMarkers.py loads and converts the XML file containing USFM codes, and provides some simple look-up functions. USFMFilenames.py accepts a folder name and searches for USFM files in that folder. All very simple, but hopefully another small step forwards.

We can use those fields for display and for keyboard input, and we have included a simple routine to create a list of all unambiguous shortcuts, e.g., we could accept Ge and also know that the user meant Genesis because no other (English) Bible book starts with Ge. But we also would like to accept Gn. So we add an extra inputAbbreviation field, where we can add things like Gnss. Again, the program will create a list of unambiguous shortcuts from this information (which would include Gns and Gn).

But why not also automatically include G as a shortcut for Genesis? Well, the problem is that G could equally stand for Galatians. But if we specified that our particular publication only contains Old Testament books, then this additional piece of information would allow the software to automatically include G as an unambiguous shortcut for Genesis. And of course, you could override by entering G as an inputAbbreviation for Genesis anyway if you desire, even for a complete Bible publication.

Ok, that’s fairly simple then. But what about 1 Timothy? It would be nice to accept 1Tim (without a space), or I Tim (using Roman numerals), 1st Tim, etc. Well the information to do this is also put in the XML file under the section BibleBooknameLeaders. This specifies everything that you will accept instead of the 1, e.g., I, First, One, etc. The software automatically handles all the various combinations for you — with and without the intervening spaces.

And then the third and final (but first in the file) section of the XML file is for BibleDivisionNames. You might want to limit a search, for example, to the Old Testament or the Pentateuch. So in this section we can specify our division names and abbreviations, and a list of the books which would be included in each division.

The XML filename includes the ISO 639-3 language code. But it also includes a qualifier, so you could have something like eng_traditional and eng_modern. Why? Well, what if you wanted to call the final book of your publication Vision instead of Revelation? It’s still English but a different system and we would need to know how you want that different bookname displayed. (Although that example is fictitious, in the Philippines where I worked it’s actually common to nickname Bible versions by the different names that the translators gave to the book of Revelation.)

So look through the data files at https://github.com/openscriptures/BibleOrgSys/tree/master/DataFiles/BookNames and tell me what other pieces of information I should still have in there. Note that I didn’t include extended book names (like The Second Epistle of the Apostle Paul to the Church at Corinth) as it seems to me that they only need to be specified in the actual Biblical material itself. I took a stab at creating basic files for French and German also, even though I don’t speak those languages, so I’m sure they’ll need some fixing. Note also that specifying the books included in a division (such as Old Testament or Pauline Letters) here does seem a little out of place (maybe it would seem more logical in the book orders data files), but since that kind of division information often relates to the cultural heritage, I’ve accepted it for now as a reasonable compromise.

There’s also some basic Python3 code to handle these data files at https://github.com/openscriptures/BibleOrgSys/tree/master/BibleBooksNames.py. It runs a brief demo so check it out and send me your suggestions and improvements.

1. We want to handle not just our own language here, but also the languages of the original texts, perhaps including Hebrew, Aramaic and Greek, plus the languages of other translations, e.g., Latin.
2. Even within one language, there might be differences. For example, many of the original texts didn’t have any punctuation (or even word breaks). So we want to be able to specify the punctuation for any publication.
3. Our standard computer locales are not likely to include the punctuation standard for Bible references and other Bible-specific uses.

Today I’ve committed the first attempt at a simple XML file for specifying the punctuation for a Bible publication. I’ve labelled my attempts as V0.40 because I’m sure that there’s going to be many more fields that we’ll need to add. And for that reason also, I haven’t yet tried to make data files for a large range of languages.

As we develop the system further, we’ll be needing ways to tell a computer program how to parse a digital text, i.e., how to determine all the different parts of the text. Another of my specific interests is also checking Bible files to make sure they are consistent ready for online or print publishing. So that requires specifying even more punctuation and formatting type details.

So as already mentioned, the current files at https://github.com/openscriptures/BibleOrgSys/tree/master/DataFiles/PunctuationSystems are really only just a starting point to work from. Let me know what else we’ll need to add.

I have found it better to separate this information from the versification information (about where chapter and verse breaks come). So the book order files are very simple — basically just an ordered list of book codes, e.g., GEN, EXO, … REV or whatever. Of course, many publications will have the same list of book contents and in the same order, so it makes reasonable sense to separate this information out so it can be easily reused.

Today I committed some book order data (XML) files which I’m aware of, although they haven’t been tested out against real publications as well as I would like. Also, there might be reason to name some of the systems better before we take it to V1.0. And, of course, it is easy to add systems which I wasn’t aware of with this initial commit. So there’s plenty of room for other people to help with their expertise here.

This is the end of the easy stuff — everything gets a lot more complex from here on in, so my blogs and code additions are likely to become much more sparse.

Any Bible organisational system has to have a way to define languages. We’ll be dealing with Biblical languages like Hebrew, Aramaic, and Greek, as well as the many languages of translations.

Many systems have used 2-letter codes like ‘en‘ or ‘fr‘ for naming languages. But 2-letter codes have a maximum of 26 squared possibilities, which equals 676 — a whole order of magnitude short of being able to represent the some 7,000 languages of the world. And since we want the Bible to reach all peoples of the world, we want to include their languages from the beginning.

So in a truly international system, we’ll need to use codes of at least 3-letters. (Yes, 17,576 codes should be enough!) And the ISO 639-3 standard, which came originally from the Ethnologue and is currently still administered by SIL gives us that.

Today I committed Python code to access the ISO 639-3 information. It’s just a little foundational step that’ll be needed later on.

[I’m not sure yet that it handles everything we need — for example Americans and New Zealanders both speak English (code ‘eng‘) but we choose, pronounce, and spell many words differently. So we might be needing to extend this language code system sometime when we get into spell-checking and such.]

But oh, that doesn’t even scratch the surface about how Bibles may be packaged! Luther considered the letter of James to be the “epistle of straw” so he placed it nearer the back of his Bible. So the book order is not consistent. Anyway, the book should really be called Jacob, not James. (It’s Greek ἰάκωβος = Iakobos and German Jakobus.) And by the way, Germans don’t use the names Genesis, Exodus, Leviticus, Numbers, and Deuteronomy (or even the German equivalent of those words)—they call them the first to fifth books of Moses. (And the Hebrews have quite different names for them again, derived from the first word(s) in each of the five books.)

Roman Catholic Bibles typically include a Deutero-canon (or secondary canon, called the Apocrypha by most Protestants) which contains several more books beyond the sixty-six mentioned above.

Some traditions (e.g., John Wycliffe’s English Bible) include the Letter to the Laodiceans in their New Testament.

Older Hebrew Scriptures may contain as few as 22 books, even though it effectively has the same content as the 39 book Old Testament, because they combine things like 1 and 2 Samuel, and the twelve minor prophets, into single books.

Some Bibles omit verses that they say are not in the most ancient manuscripts so they might go directly from verse 20 to verse 22 in a certain chapter. (See if your Bible contains Matthew 17:21, for example.) Other Bibles add extra chapters and verses which they get from the ancient Greek translations of the Hebrew books. And some of those chapters may be labelled with letters (like A, B, and C) rather than numbers.

And talking about chapter and verse numbers, they are not found in the original manuscripts but were added at different points in history. And different people in different cultures divided the texts in different ways. So, for example, in the Psalms in a Hebrew Bible, the text A Psalm of David might be considered verse 1 and what I’ve always thought of as verse 1 is called verse 2.

Modern translations tend to divide the texts into paragraphs, and these breaks (and their corresponding section headings) also might be placed in different locations according to the best judgement of the various translators.

Many modern Bibles contain cross-references, e.g., a quote in John 1 might refer back to Genesis 1. But the code “Gen. 1:1” might look different in another language where they use a different name for the first book of the Hebrew Scriptures, and a different punctuation character for the chapter/verse separator, so it might have “1 Mos. 1.1” in the cross-reference instead. Also, letter suffixes may sometimes be used in Scripture references, like “Mat. 28:3a”.

Most of the hundreds of available Bible software applications have had to address these issues, and most of them have developed their own unique in-house systems to handle them. In many cases, especially for those of us in the often monolingual English-speaking world, the cross-cultural ramifications weren’t really understood at the beginning and so extensions were tacked on later to try to handle these international complications, sometimes leading to systems lacking in elegance or universal application.

I am going to slowly feed my work into the Open Scriptures BibleOrgSys repository over the next few weeks. The essence of my work can be found in XML data files, but I’ve also produced Python scripts that test and exercise (and even export) the XML data to demonstrate how an application might use the data files.

So this work is my attempt to develop a system that is multilingual, multinational, and multicultural from the beginning—to pull all of these various Bible organisational systems into one place. And although I’m an English-speaking Protestant Christian, it tries not to assume that every Bible-related publication in the world is based on any of those particular characteristics. And then it is made freely available to be used by others, and even to be extended by others when areas I didn’t cover adequately are discovered.

For those with experience in this area, I look forward to your comments, and even better still, suggested improvements. I will label the submitted files V0.5 to suggest that I might have done half of the research and work in setting this up, but there’s plenty of room for your input and wider experience to take us to V1.0. You can view my work on Bible books codes at https://github.com/openscriptures/BibleOrgSys.

Open Scriptures API: Distributed model for interlinked text development

In the continuing efforts at Open Scriptures to produce an open platform for the development of scriptural data and its applications, work continues on the most fundamental layer: the representation of the scriptural text. This talk will look at a normalized way to store text in a relational database as tokens and structures, and also how such a text can be versioned as it undergoes continuous improvements (e.g. making a translation or creating a critical text); as the text can exist in multiple revisions/editions, it can also be branched so as to introduce variant or alternate readings; it can also be forked to introduce a new derivative work linked back to the original. (Much of this will be applying lessons from the Git distributed version control system.) In addition to links between text editions, the representation of links between different texts (interlinearization) will be examined at the level a single scripture server and also in the context of a distributed network of scripture servers which all have texts that are potentially undergoing development and how the interconnections between the texts can be maintained to enable scriptural linked data.

This is a continuation of last year’s talk, Open Scriptures API: Unified Web Service for Scriptural Linked Data.

First of all, md5 is broken and deprecated, due to possible collisions (two different pieces of data can result in the same hash). Since we will be dealing with millions of tokens, we decided not to test our luck, unlikely though a problem may be. SHA-256 has no known collisions, so we decided it was best to use that algorithm.

SHA-256 is implemented in Python’s standard library hashlib, so that is good. For exapmle:

>>> import hashlib
>>> hashlib.sha256("Hello world!").digest()
'\xc0S^K\xe2\xb7\x9f\xfd\x93)\x13\x05Ck\xf8\x891NJ?\xae\xc0^\xcf\xfc\xbb}\xf3\x1a\xd9\xe5\x1a'

Needless to say, such a digest would not be very good for use in a RESTful URI scheme. So, hashlib also offers a hexadecimal option:

>>> hashlib.sha256("Hello world!").hexdigest()
'c0535e4be2b79ffd93291305436bf889314e4a3faec05ecffcbb7df31ad9e51a'

That is still not the best, since that makes for a very long string. So, we have the option of using base64 encoding:

>>> import base64
>>> base64.b64encode(hashlib.sha256("Hello world!").digest())
'wFNeS+K3n/2TKRMFQ2v4iTFOSj+uwF7P/Lt98xrZ5Ro='

That is shorter, but it includes the “/” character, which is a no-no for URI design. Luckily base64 includes a function for this exact purpose:

>>> base64.urlsafe_b64encode(hashlib.sha256("Hello world!").digest())
'wFNeS-K3n_2TKRMFQ2v4iTFOSj-uwF7P_Lt98xrZ5Ro='

So that is safe for URIs, but being case-sensitive and having ambiguous characters makes it not the best for working with. So, base32 to the rescue:

>>> base64.b32encode(hashlib.sha256("Hello world!").digest())
'YBJV4S7CW6P73EZJCMCUG27YREYU4SR7V3AF5T74XN67GGWZ4UNA===='

There you have it: shorter than hex, and easier to work with than base64. If we end up using cryptographic hashes an token unique IDs, I’m pretty sure this is how we’ll do it.

The discussion around using cryptographic hashes as unique identifiers in our models is ongoing. Essentially we need to decide how best to make a unique hash for each token. Please join us in #openscriptures on irc.freenode.net with any input.

Open Scriptures API: Unified Web Service for Scriptural Linked Data

By Weston Ruter

Video: Vimeo
Slides: Keynote
Audio: MP3

Using Pinax and Django For Collaborative Corpus Linguistics

By James Tauber

Watch video on Vimeo, and see also his talk on A New Kind of Graded Reader.

More audio, slides, and potentially video of additional talks will be posted on the BibleTech speakers page.