Category Archives: asciidoctor

How to create your own OpenOffice backend w/ @asciidoctor /cc @kubern @ysb33r

asciidoctorMy work on creating an asciidoctor backend for Packt Publishing (i.e. OpenOffice) seems to have caught several people’s attention. Today I spotted an exchange of tweets asking about OpenOffice in general. I figured a more detailed approach to doing that would better fit in a blog posting.

Starting with the basics

For starters, this whole thing is based on creating an FODT document, not an ODT one. The “F” stands for Flat and it is basically an XML document. Your mission, should you choose to accept it, is to create an XML document with all the right parts. That’s the reason my backend is written in SLIM.

SLIM is basically XML with every bit of unnecessary stuff removed. No closing tags. No angled brackets. Nested elements are denoted by indentation, hence giving it a more YAML-like feel. Checkout the following fragment for generating different section levels.

Working backwards

My recommendation is to start from a document you know and love. I started with Packt’s template they sent me. I performed a “Save As” and picked “Flat ODT” to get it into XML format. I then wrote a sample asciidoctor document with all the things I wanted. I suggest you start small and not get into complex layouts (yet). Remember: Minimum Viable Production (MVP) is the key to short, iterative success. From there, I started running my backend against this thing and comparing it with the outcome.

Got xmllint? If not, you should. Your output will be compressed XML. xmllint is the key to viewing your FODT output. Got giant monitor? .It can help dealing with stuff scrolling off the screen as you inspect your handiwork.

A key thing to understand. Asciidoctor splits up each “element” of asciidoc into a separate fragments and looks for the corresponding stuff in your backend. Got a “section”? Look for slim/packt/section.fodt.slim.

  • slim is the template engine
  • packt is the name of the backend (hence all your fragments will be there)
  • section.fodt.slim is convention-over-configuration for generating an FODT-styled chunk of XML everytime asciidoctor hits a “section”.

The top level thing for you document is, duh, document.fodt.slim. And this is also where you’ll find the root FODT XML <office:document>.  Way at the bottom is a common pattern, “=content”. This says, continue processing further asciidoc tokens and insert results at this level.

Iteration is the key to success

To run your backend against your asciidoc input:

asciidoctor -T /path/to/your/backend-root/slim -b packt your.adoc

My backend doesn’t cover every possible element of asciidoc because I didn’t need them. With asciidoctor, you can start with just document.fodt.slim & helpers.rb and when it hits an unknown token, it will tell in simple terms. Go and code <foo>.fodt.slim, and run again. Eventually, you can piece out your entire document. And in the process, you’ll become quite fluent with both FODT and asciidoctor.

Don’t give up!

You will undoubtedly hit some bumps. I sure did. One was getting my code fragments to apply a different style to the last line of any block of code. I probably invested two weeks of effort in this entire backend while writing Learning Spring Boot, but the payoff was enormous. If you are trying to convert a giant slew of documents, try to avoid after the fact edits. And if possible, get the core stuff working, like sections, paragraphs, bold/italics, bullets and admonitions. That stuff really motivated me to push forward.

Oh yeah…

Last time I checked, asciidoctor has special processing rules for double underscrores (__). Several of my style names from Packt had them, and it caused havoc. Once I tracked it down, I simply did a global replace, reducing them to single underscores. DON’T USE DOUBLE UNDERSCORES FOR ANYTHING!

Good luck!

Valuable things I have learned while writing three books for @PacktPub

I recently had a friend of mine ask me about the viability of writing a book for his open source project. He had questions about how to submit a proposal as well as the money involved. I wrote him a detailed response, but decided to post some of my lessons learned here.

Don’t write a book for the money

This is definitely a worn out expression. I have seen people get this starry gleam in their eye, Sorry, but that only happens when you write a best selling, fictional trade novel. Not everyone can be J.K. Rowling. In fact, there are actually a very small number of authors who make it big. The real name of the game in writing is to write LOTS of books, such that over time, people that like your material end up buying all your titles.

When writing technical books, you still need to sell a bunch to really make any money, but we are forever cursed by books going out-of-date in just a few years. Trade novels last a lot longer.

That may sound a tad depressing. So why write book? Because it can open doors you never knew would happen. When you are a published author, it can impact interviews, speaking engagements, consulting opportunities, and other things. It is always a major +1 to your CV to have written a title. Even more if you get several under your belt. It also increases your overall name recognition and personal brand.

Understand what you’re getting into

Before expending any effort in writing, I made sure we had a contract in place that we both agreed to. I’ve read article where people keep writing chapter after and yet have not signed an agreed upon contract. I don’t work that way. You are taking on risk by doing that. Until a contract is in place, the publisher can bring onboard other writers.

I have historically preferred to write my titles by myself. It lets me control the content 100% and not have to deal with interacting with another author, one who might/might not be as motivated as me. As a nice side effect, it also ensures that I get all the money. Since we’re already talking about a small pile of money, it feels counter production to split it up amongst two or three people.

Packt’s contracts include an option of first refusal for your next two titles. I understand Packt’s desire to get first dibbs on future work. It make business sense, so I had no issue with this in my first two titles. But on the third one, I had decided they had made enough money on me. I asked that this clause be removed to liberate me for future works. I have no hard feelings nor have I felt any malice from anyone at Packt.

Be sure to understand that the contract includes deadlines and exactly how much money you’ll be paid and when. If you sign the contract, you are agreeing to everything stated. Understand what you are agreeing to.

It’s your title. Act accordingly

Every publisher out there will probably tell you how glorious their marketing and sales team is. But for you to write a book and make no mention on Twitter/FaceBook/Google+, nor to write a peep on your blog site is ridiculous. You should be blogging all the time as you write and beyond publishing. Don’t assume that your publisher will automatically sell 1000 copies of your book and be fired up to order another batch!

Another aspect of “acting accordingly” is to remember this will reflect first and foremost on YOU. If you have a certain voice or a style of presentation when you speak at conferences, then by all means, try to write that into your manuscript. Your editors may try to edit that out. If you give them the upper hand, they can reduce your effuse writing into boring humdrum. I’ve seen editors edit out some of my sassy writing. My response? I edit it back in! They don’t usually push back too hard.

It’s important that the publisher have a certain level of quality. But there needs to be enough of YOU in whatever you write that makes it stand out.

Go and read other people’s technical books. Part of this inspiration for me came from a couple other technical books I read. The style these author’s wrote knocked me out of my chair. I LOVED the small jokes, clever class names, and funny material they injected into their writing. And it told me I could do the same, as long as it wasn’t offensive or obnoxious. Again, my editors pushed back, but I paid keen attention to EVERY edit, and undid several of these. I wanted my book to be fun.

Making your book fun is no one’s job but yours.

Take credit for your work – give credit to others

In my latest title, I started each chapter with a quote from twitter. “Spring Boot” was so hot, that I started collecting tweets as I wrote. I would launch a chapter, and include a link to that user’s twitter handle. Their comments were inspirational. I hope they picked up new followers.

At the same time, I gave a presentation that pre-dated release of my book. I mocked up a logo of the book with the title in it, and included that in my profile slide at the beginning. This was a PR moment I wouldn’t pass up. I wanted everyone in that audience to know a book was coming, and hopefully they all bought a copy!

That same week, my colleagues and I trekked to a JUG meeting. Someone in the group asked, “Is there a book on Spring Boot coming?” Everyone turned to me, and I happily gave them the estimated release date. More free PR. I also was sure to cc more than a handful of tweets to that group’s twitter handle. Hopefully they each bought a copy! (Not likely, but one can dream, right?)

Finally, I wrote my last title using asciidoctor thanks to Dan Allen. This was by far the best option. It allowed me to focus on quality content and not idiotic Word Processor issues.

Knuckle down and write

Okay, all these suggestions up to this point are nice and all, but it’s all for naught unless you put in the hours to actually write your book. I read someone’s blog article where they thought the deadlines were crazy since they only planned to write on the weekends.

Sorry, but I don’t have much sympathy there. I wrote each of my books while we also had a newborn baby. I quickly adopted the style of writing every night after everyone had gone to bed. This way, I could focus with no distractions, and then take the weekend off. Writing every night from 10pm-midnight can be exhausting when you mix it with feeding a baby at 2 AM, but this ensured that I got my book finished and out the door. It also cut down on the risk of my technology moving away from me too quickly and making the material invalid, or avoiding massive rewrites due to mega-changes.

Another thing you have to be ready to do is carry your book all by yourself. You can’t depend on certain reviewers to fix things for you. Instead, you need to be ready to proof read in the event your book reviewers don’t actually give you any feedback until the last minute. It’s sad, but this still reflects on you.

There were times when I hit bugs in my sample code, and had to simply pack up for the night. There were other times when I had no choice but to work an extra two hours to hammer out a problem that no one else could help me with, either in my asciidoctor backend, or in my sample code. After all, no one was going to pick this up and fix it, right? Leaning on pull requests, etc. is no excuse for not writing your book.

I demoed Spring Social GItHub in my latest book, and had to go to press using version 1.0.0.BUILD-SNAPSHOT because the project lead was simply too backed up to fix the milestone release already published which broke my book’s sample code. Them’s the breaks.

So by now, if I haven’t smashed your dream of writing a technical book too badly, I wish you only the best in going out, submitting a proposal, and snagging a contract.

I owe so much to @russmiles!

Russ's shipment of Learning Spring Boot, ready to distribute to conference attendees

Russ’s shipment of Learning Spring Boot, ready to distribute to conference attendees

Russ is a good friend of mine I met back in 2008. He was working for SpringSource at the time as a consultant. He had spotted my open source project, Spring Python, and invited me to make it an official Spring Extension. I was quite excited! Along the way, I had this crazy idea of writing a book about it. I had seen so many other Spring books that I wondered if I could do the same.

When I bounced this ridiculous idea off Russ, he immediately told me to go for it. Having written a couple books, he had suggestions on how to pitch it, who might be willing to publish something about what was frankly a niche project, and also brought on board Sylvain. Thus was born Spring Python 1.1.

Throughout the writing process, Russ provided a constant stream of useful feedback and assistance, a veritable mentor by any definition. (The irony being that I’m older!) Fast forward to now. My third technical published, Learning Spring Boot. This time around, I negotiated different options in my contract, I took the liberty of writing it in asciidoctor, and I’ve journaled about the writing the whole time to give potential readers an insight into what they can expect.

When my editor asked about ideas for marketing, I immediately suggested they send Russ a truckload of copies to hand out. He travels to all sorts of conferences. Naturally Russ leaped at the chance. You can see the picture of his shipment at the top of this article.

I have mentioned before that I picked the cover art for Learning Spring Boot to symbolize how the power of Spring Boot is built upon a strong network of roots underneath the forest. In that case, I was speaking about the Spring Framework underneath Spring Boot. Regarding my ability to write, Russ has been a strong foundation for my ever growing reach into the written world. What more can a fellow software geek ask for?

Cheers mate!

That’s a wrap! “Learning #SpringBoot” ‘s 1st draft has been submitted to @PacktPub

learning-spring-boot-ch-5Last night, I worked from 9:30pm until 2am on Learning Spring Boot. Whew! It was tough work, but I needed to pull it across the finish line. I had the code lined up quite nicely for the entire chapter. I simply need to tell the story of how all this stuff worked together.

As I’ve said before, security by itself is complicated. There is a lot of fine details involved. When someone first starts reading the reference docs of Spring Security, it can get real intimidating real fast. That isn’t Spring Security’s fault. It’s inherent to the nature of locking something down except for the right people in the right context with the right protections. “right” and “wrong” can be very hard to define in just a couple sentences.

I tried to walk through what is happening, provide a little “why is this happening” and finally leave some links for those that want to dig a little deeper and understand the incredibly complex stuff that Spring Security is ACTUALLY doing for them. The protocols Spring Security picks up and handles are quite complex, and when you get to the end, I hope the reader has a strong appreciation for how simple Spring Security has made it to give the developer control while still protecting system integrity.

With all that said, I breathed a big “whew!” as I bundled up this chapter’s manuscript and emailed it over to my editor at Packt. I have already started receiving feedback on the other chapters from my review team. These people are sharp! They really know Spring and have spotted some sloppy mistakes on my part. After I parse all their comments, this should be one well honed book!

asciidoctorI also have truly enjoyed the development process of this book. Writing it in asciidoctor has let me focus 95% of my effort on the content. I work with an IDE open so I can tweak the code, run it, tweak again, and then commit the changes. My manuscript simply includes code files and fragments, and I layer it with some prose to explain what is happening and why it’s happening. Here and there I put tips and notes to answer questions I can foresee.

Then on the command line, I constantly regenerate an HTML rendering of the book. I read through it in chunks and polish it up. After all is done, I finally open up the LibreOffice output and and walk through to fine tune bits of code that overrun the width of the page. By staying away from the word processor as much as possibly, the integrity of this book’s content has been kept at what I feel is tip-top shape.

docbook…what a piece of junk!

lukepieceofjunkIn the past week, I have focused on porting the Spring Data Evans release train to asciidoctor. It has been joyous and torturous. The joy is moving away from docbook and towards asciidoctor. The torture is dealing with all of docbook’s idiosyncrasies and the wonderment at how someone could build such a tool in the first place.

First of all, XML is a highly nested data structure. The nesting is used to convey semantics, but XML was originally designed as a machine-to-machine communication medium. It was indicendtially written using ASCII so that devs could debug the communications more easily along the way. Using it to create documentation systems is ridiculous.

docbook-whitespaceIf you look at this fragment of docbook, you can see all the wasted space based on standard XML formatting. Reading and maintaining this is an eyesore and the source of carpal tunnel.

asciidoctorLooking at asciidoctor is much more pleasing. There is no cognitive overhead of angled brackets. No extra indentation. And the results are easier to visualize. The semantics are so much clearer.

But that isn’t the only thing that has been a hassle. To do this migration, I wrote a SAX parser that properly pushes and pops as elements are started and finished. In essence, as you start a new element, a new state needs to be created an pushed onto a stack. When that element finishes, you pop that state off the stack, and properly append it to the previous state, i.e. the enclosing XML token. I had this working smoothly. Along the way, if you hit any characters between the XML entities, they need to be appended to the top of the stack’s “chunks”.

But docbook has so many awkward situations, like <section> <title>blah</title> <para> <section> <title>blah</title>…., that would lead to a level 0 header followed by a level 2 header. Where is level 1? Nowhere to be found and up to me to clean up.

Also docbook has SO MANY different tokens that really expresses the same thing. For every different element type, I created an AST node to represent it. But when it came to <code>blah</code>, <interfacename>blah</interfacename>, and <literal>blah</literal>, I created only one, Monospaced. These all basically get converted to the same thing: `blah`, i.e. some text wrapped in back ticks.

Other stuff is simply impossible. I did my best to handle tables, but too many quirky things can go wrong. So I did my best and then hand edited after the fact. Callouts are worse. Asciidoctor makes callouts awesome. But getting over the hump of docbook’s format is such that I don’t even try to convert. I just stuff the raw AST content in the output and warn “DO THIS PART BY HAND”.

Given all this, porting the whole documentation of Spring Data to asciidoctor required me to invest three days of effort purely at building this script. I actually rewrote the script after Day 2 because I had made a fundamental flaw. But I met my objective of migrating all of the Evans train by this past Monday.

But the pay off of writing the script properly has been great. I hadn’t written a SAX parser like this for ten years. It brought back keen memories. Thankfully, using Spring Boot CLI + Groovy made it a pleasant, scriptastic experience.

 

A new project in my spare time…helping my nephew create games with #Python and #Pygame

My nephew expressed interest last year in creating games and programs on his computer. I bought him a super cool book, Making Games with Python and Pygame. He’s been busy with other things, but today I got a call from him asking how to set up Python.

I was driving so I couldn’t get very far. But the pure fact that his parents have a Mac made me smile. We could speak on the same wavelength. I promised to send a follow up email with some HOWTO steps.

The first thing I did was reach for Asciidoctor and start writing up a handful of steps to get started. I had to really dial up the thinking, because I wasn’t writing for a pro developer. I was writing for a kid that doesn’t know any of this.

I also decided to host it all at https://github.com/gregturn/so-you-want-to-learn-python as the place to manage everything. After getting that fixed up, I realized I could pipe the generated HTML into a gh-pages branch, and host the polished stuff at So You Want To Learn Python.

Should be enough to get the ball rolling. Waiting eagerly for a response!

Chapter 2’s first draft is roughed out

ch2_fodtLast night, I finished up the last bits of chapter 2’s first draft. I like to knock out a chapter in the rough and then go through it, applying bits of polish. For me, it’s always easier to polish something that already exists.

My biggest concern was the fact that Spring Boot keeps taking away space filling code! So instead of creating some boring toy app, I had to think up something bigger. I figured an app that scans open issues from multiple GitHub repositories (part my upcoming webinar on creating management apps with Spring Boot) would be good enough. Wrong!

This chapter’s demo app has become way too small! Ever hear of that? Our apps usually are too big, especially for the medium of print. I tried to fill up the allotted space with deploying to one of the fastest growing PaaS’s out there: Cloud Foundry. There are over thirty members of the Cloud Foundry Foundation, so showing how über easy it is to push your Boot app up to a cloud should eat up space, right.

Wrong again! Well this has forced me to roll up my sleeves and actually write some pragmatic support scripts that load the system down and gather metrics at the same time….using Spring Boot!

So after slaving away at creating useful code, deployed to a useful platform, with useful support, I figured any more content and this chapter would explode in practical density. I saved my last changes and decided its time to now go through and polish things up. There’s always an opportunity to add another chapter later on, right?

It’s nice to read through the manuscript and see if it flows. Am I going to fast? Too slow? Sometimes its a mix of both. Given that this will be THE first Spring Boot to hit the proverbial shelves, I want it to be a good reading experience. I also ponder what questions the reader might be thinking, and try to insert pull-outs answering them.

I’m also incredibly happy that I was able to get a functional Asciidoctor backend written AT THE SAME TIME. Don’t worry, I drink plenty of coffee in the morning to recover from my total lack of sleep. 🙂

Stay tuned for future developments!

The value of software communities is greater than ever

ch2_fodtAs I continue working on my new book, Learning Spring Boot, I have really noticed the value of communities today compared to when I became a professional software developer back in 1997.

Back then, there were news groups and mailing lists, but not the same indexing of EVERYTHING, no Google Groups (which just made searching/working with news groups MUCH easier), no panacea of discussion forums, and no proliferation of blog sites. These things existed, but you wouldn’t find so many, and so dedicated (like trying to write a query for your trade novel, one of my favorites). And open source hadn’t taken root like it has now. Your prospects of finding an open source project with a thriving community to serve your needs were MUCH less likely.

So how exactly did we survive without such a rich community? Sometimes I’m mystified. Back then, I would comb the book store for help. It also meant I had to hunker down and really go through code myself to figure stuff out. What’s that you say? You think that I’m whining about having to actually WORK back then? I worked then, and I work now. It’s just different.

Today, perhaps there are more people willing to help you get over a hump. Back then, getting help meant walking across my cubicle hallway and asking the right team guru questions. Over time, I would pick up my own set of skills and quirks. And others recognized that. Eventually others would come to me asking for assistance.

Back then, I knew someone that NEVER deleted emails. I didn’t comprehend such a concept. Today, I’m a tried-and-true believer in keeping everything. Knowledge is power, and having my entire history of communication available has been useful. In fact, when I finally moved my personal email off the telco provider I’d used for years, I searched for someone that could handle migrating all that email to my new account.

Today, it seems impossible to develop anything in a vacuum. I tap twitter, discussion forums, and other places. The latest incredibly powerful social tool for interacting, getting feedback, and making substantive progress if GitHub! Git is a powerful tool for version management, but you can’t realize its true value until you work on a community project with forks, issues to discuss things, and ultimately pull requests.

I launched my book writing endeavor by blogging and tweeting what I was up to. On my previous book, I got enough attention that I was able to shift off a tool that had fallen into disuse and pick something better. I even got quotes from project leads into my book.

This time, I attracted the attention of both Dan Allen, the project lead for Asciidoctor, as well as an editor from Packt Publishing. I am building a new Asciidoctor backend for Packt, and this might be the track that makes it mainstream. Dan quickly helped me get going on something that I couldn’t make heads or tails of when I read it by myself. If Packt bites at somehow making this backend an officially accepted medium, the lives of MANY future authors may be made simpler while also improving the content of future titles. (That applies to my own future writings!)

To paraphrase Hugh Williams, Pivotal’s new SVP, skills are fleeting. The knowledge you are using today will probably be totally replaced ten years from now. But what’s critical is judgment, ability to learn, and willingness to interact with others. And nothing serves that better than a strong community.

Goodbye #AsciiDoc, hello @Asciidoctor. Tnx @mojavelinux!

asciidoc_numbered_listI just got my last few tweaks into my Asciidoctor Packt backend. It was enough to drop the venerable AsciiDoc and replace it with Asciidoctor.

Most notably, I fixed the images so they would be centered, and text would not wrap around. I also centered the [[Layout]] note that always is placed below. It looks perfect!

I can say this: working with an asciidoctor backend is gobs simpler than that asciidoc backend. It was brittle, hard to decode, and most of the time I was editing a monolithic file. With asciidoctor backends, each logical fragment is split off into a separate file.

In each file, I was using slim, a very terse way of writing templates. Given that the output is FODT, i.e. XML, slim is pretty neat. You get to express XML structures without the overload of reading angled brackets or having to write the closing tag.

But none of this would have been possible without Dan Allen’s direct involvement. I tried to get started on my own, but it was a no-go until Dan stepped in and submitted a pull request with an initial cut of a backend along with some critical steps to use it. To top it off, we were able to work back and forth, either through more pull requests, or through a discussion thread, to resolve other critical issues. At the end of the day, Dan is one rock solid community builder.

This already opens the door to using options unavailable before hand. The biggest one is how I want to extract fragments of source files into my manuscript. There is still a pending pull request to slightly alter the section format of the manuscripts, but this is structural and doesn’t interfere with my writing.

Cheers!