A converter between yWriter and novelWriter

[peter88213]

As I announced earlier, I wrote a Python script that converts projects between yWriter and novelWriter in both directions. Now the first alpha release can be downloaded.

For more information, see the project homepage with description and download instructions.

Because of the different structures and features of yWriter and novelWriter, there are of course some limitations. All this is described in the instructions for use.

Have fun with it!
Peter

[vkbo]

Very nice! I'll have to try it out. I haven't used yWriter in a long while, but as I've mentioned it's one of the original inspirations for making novelWriter. As a Linux user, yWriter is a bit inconvenient to use.

I should add this to a section in the README and docs for tools to use with novelWriter.

[peter88213]

When trying it out, please be sure to use the latest version, since it's still alpha state. However, I succeeded in converting my current 40.000 Words work in progress in both directions.

As for running yWriter on Linux, WINE is the only way that works at the moment. First let winetricks install dotnet 3.5, then install the latest yWriter5 release (.exe will work).

[vkbo]

Just a heads up that I will be changing the file format a bit in the next release. The updated project XML file looks like this: nwProject.nwx.

The changes are explained in #993.

They are quite simple. Since the <item> node had a lot of child nodes, I converted nearly all of them into attributes of the <item> and <name> nodes, and added a new <meta> node. If you look at the diff of the NWItem class that does the reading and writing, you'll see that it is a fairly simple change: novelwriter/core/item.py.

For the converter, I bet you can drop the <meta> node entirely as it doesn't contain any information I suppose you can transfer between the applications anyway. The type, class, and layout settings are now attributes of <item>. The <name> node is the same as before, but the exported flag and the the user-defined status value are attributes of <name>. Most values can be omitted, and will have a default setting.

The format change has been merged to the main branch now, but is not included in the 1.6 release I just made. It will not be out in the wild until 1.7. The file format has changed version from 1.3 to 1.4.

The reason for making this change is that the new format is a lot more compact and I plan to make changes to novelWriter to support multi-novel projects, which will mean the number of item nodes may grow a lot in large projects. I haven't changed how items are stored since the first versions of novelWriter, so the list of nodes has been growing quite large because of the verbose format.

[peter88213]

In my case I have defined the scene content as a property; the setter calculates the word count on each change. With Python it looks like this:

    @property
    def sceneContent(self):
        return self._sceneContent

    @sceneContent.setter
    def sceneContent(self, text):
        """Set sceneContent updating word count and letter count."""
        self._sceneContent = text
        text = re.sub('\[.+?\]|\.|\,| -', '', self._sceneContent)
        # Remove yWriter raw markup for word count
        wordList = text.split()
        self.wordCount = len(wordList)

But may not be enough to have a living word counter in the editor.
As for novelWriter saving word counts, etc., I had suspected it was mainly for file previewing.

[vkbo]

The word counter in novelWriter is not quite that simple. See the code:

novelWriter/novelwriter/core/index.py

Lines 823 to 897 in 011291f

	def countWords(theText):
	"""Count words in a piece of text, skipping special syntax and
	comments.
	"""
	charCount = 0
	wordCount = 0
	paraCount = 0
	prevEmpty = True
	if not isinstance(theText, str):
	return charCount, wordCount, paraCount
	# We need to treat dashes as word separators for counting words.
	# The check+replace approach is much faster than direct replace for
	# large texts, and a bit slower for small texts, but in the latter
	# case it doesn't really matter.
	if nwUnicode.U_ENDASH in theText:
	theText = theText.replace(nwUnicode.U_ENDASH, " ")
	if nwUnicode.U_EMDASH in theText:
	theText = theText.replace(nwUnicode.U_EMDASH, " ")
	for aLine in theText.splitlines():
	countPara = True
	if not aLine:
	prevEmpty = True
	continue
	if aLine[0] == "@" or aLine[0] == "%":
	continue
	if aLine[0] == "[":
	if aLine.startswith(("[NEWPAGE]", "[NEW PAGE]", "[VSPACE]")):
	continue
	elif aLine.startswith("[VSPACE:") and aLine.endswith("]"):
	continue
	elif aLine[0] == "#":
	if aLine[:5] == "#### ":
	aLine = aLine[5:]
	countPara = False
	elif aLine[:4] == "### ":
	aLine = aLine[4:]
	countPara = False
	elif aLine[:3] == "## ":
	aLine = aLine[3:]
	countPara = False
	elif aLine[:2] == "# ":
	aLine = aLine[2:]
	countPara = False
	elif aLine[:3] == "#! ":
	aLine = aLine[3:]
	countPara = False
	elif aLine[:4] == "##! ":
	aLine = aLine[4:]
	countPara = False
	elif aLine[0] == ">" or aLine[-1] == "<":
	if aLine[:2] == ">>":
	aLine = aLine[2:].lstrip(" ")
	elif aLine[:1] == ">":
	aLine = aLine[1:].lstrip(" ")
	if aLine[-2:] == "<<":
	aLine = aLine[:-2].rstrip(" ")
	elif aLine[-1:] == "<":
	aLine = aLine[:-1].rstrip(" ")
	wordCount += len(aLine.split())
	charCount += len(aLine)
	if countPara and prevEmpty:
	paraCount += 1
	prevEmpty = not countPara
	return charCount, wordCount, paraCount

In any case, the information saved in the project XML is just to be able to quickly populate the tree without having to run the counter on the files. So the value is just a cached number. With the meta data there, no text files are loaded at all during start-up. This makes start-up a lot faster for large projects. The numbers are only duplicated into the project XML because the index isn't guaranteed to exist on start-up. When a document is being actively edited, the counts are re-computed every 5 seconds by a background thread.

[peter88213]

Basically, we use the same word counting algorithm: len(text.split()) Because yWriter separates metadata from content from the start, it's a bit easier overall. Nevertheless, the word count of my converters almost never matches that of yWriter itself, just as each application provides different numbers (especially at NaNoWriMo, this is a constant topic of discussion).
If you want to be very precise here, you have to agree, for example, whether abbreviations like "It's" or hyphenated composites in different languages should count as one or two words. Then you would also have to look for true and false apostrophes and so on. Actually unimportant, I think, but with 50,000+ words the differences do hurt one or another.

[vkbo]

Yeah, my code mostly just filters out formatting syntax. The rule I use for splitting words is indeed just whitespace, but I do also include dashes as word separators. I just replace them in the text before running the split, so both American style long dash pauses with no spaces, and short dashes used to define ranges and such, count as two words. Hyphenations count as single words.

The problem with getting too nit-picky about word boundary definitions is of course that it depends very much on the language. In Norwegian, as with German, we have a tendency to make words by combining multiple other words. Generally without hyphens.

Personally, I think character count is a much better and more consistent metric than word count. My main semi-professional writing experience is with essays for publications anyway, where character count is the norm. At least around here.

In any case, on the single document level this doesn't matter as much. I'm considering adding some text analysis tools on the compiled manuscript text though, where processing time is less of a concern and there is a user interface where the user can be allowed to make some decisions about how things are counted. A tricky one her is of course syllable counting. A topic I've looked into before.

[peter88213]

Since novelWriter can export the ODT format anyway, Libre/OpenOffice can already take a lot of work off your hands, since they have a sophisticated word counting algorithm as far as I can tell.
Traditionally, it's still the number of pages that counts in Germany, and that ultimately also essentially determines the printing costs.
Therefore, my yWriter to ODT exporter generates standard pages with 10-pitch fixed width font and 30 lines with maximum 60 characters. Then for Openoffice I have a "Style Switcher" extension that lets you quickly switch between document styles if you want to see something different on the screen or in the printout.