My Kindle Mocks My To-Read Shelf: Machine Learning, Bestsellers, and the Future of Publishing

Standard

This short (>8min) Vox video on machine learning is terrific. It’s a fascinating look at the way work is being automated.

It also reminds me of last month’s post about the academics who created an algorithm to analyze books to see if they’ll become bestsellers or not. Brief summary: they subjected thousands of books to several kinds of analysis in order to identify traits that the bestsellers had that the non-bestsellers did not. They found nearly 2800 distinct differences.

The algorithm couldn’t create a bestseller, and in their book the academics were clear the technology was a long way off, if it was possible at all.

The video above explains why that is, and why the software’s ability to teach itself is so interesting. Recommended.

Since that last post, the academics who developed the algorithm and wrote the book have opened a consulting service. Of course, right? It’s the natural next step. As an author, I guess I’m supposed to find this threatening/a sham/the end of literature, but I don’t. It’s just information. The only real question is whether it’s good information.

I won’t be worried until the day editors stop reading manuscripts my agent sends them unless they’re accompanied by an Archer-Jockers Score(tm). And I don’t see that happening in my lifetime.

But no, seriously, that’s an interesting video up there.

As I write this, The Twisted Path has eighteen reviews on Amazon. My short fiction collection, which includes a 20P novelette, has eleven. It would be extremely useful if those numbers could be boosted to twenty-five. Fifty would be even better. Amazon has algorithms of its own, and works with a number of reviews that pass a certain threshold get more prominent placing in search results.

It’s all pretty opaque, but what it boils down to is more reviews=more visibility. If you read and enjoyed either of those works, please consider dropping a review for it.

SPFBO, The Self-Published Fantasy Blog Off

Standard

Have you guys heard of this?

Author Mark Lawrence, in an attempt to help self-publishing authors publicize their work, created a Self-Published Fantasy Blog Off. The system is simple: He recruited ten reviewers with blogs, solicited 300 fantasy novels from self-published authors, and distributed them. Each reviewer picks one out of their 30 to move to the finals. Then the reviewers choose a winner.

The Way into Chaos is one of those finalists.

The winner gets an award, but most importantly, a publicity boost, which is a big hurdle for self-published work. Last year’s winner also landed a publishing deal with Orbit.

So, if you’ve been thinking you’d love to try some indie fantasy but don’t know where to start, snag one of these finalists (right after you read mine).

Speaking of snagging one of mine, did you know that I’m trying to revive my Twenty Palaces series with a new novella that picks up where Circle of Enemies left off? Grab a copy today.

Seeing the Forest for the Algorithm: a Review of Past Edit Notes and a Hard Truth

Standard

In which the author makes an embarrassing confession

One of my little secrets is that, in between projects, I’ll sometimes read a book about writing. It’s always useful to reinforce the basics, and seeing how other writers approach the blank page gives me insights into my own work.

Sometimes I get the impression that I’m supposed to be past all that, but I’m not. I’ve never really felt that I’ve mastered this craft. Some aspects of it, maybe, but I still struggle.

And since I’m brainstorming something new, I took another writer’s casual mention of his favorite book about writing, Stephen King’s On Writing, and borrowed it from the library. I had barely started when I heard a discussion of a different book over the radio. You can listen to that here, if you really feel you have to. It’s not an interview with either of the authors, and the interviewee’s Wired article is more interesting and informative with fewer dopey questions.

The book is The Bestseller Code by Jodie Archer and Matthew Jockers. Maybe you remember when it came out last year, or maybe the title is enough to guess what the book’s about. The authors created an algorithm to analyze a variety of modern novels, then ran all sorts of books through them: bestsellers, non-bestseller, midlist books, self-published, the whole deal. The algorithm noted the differences, then sorted out the ones that were strongly predictive of bestsellers. According to the authors, their “bestseller-ometer” was capable of predicting whether a book would be a bestseller with 80% accuracy.

It’s correlation, sure, but the authors found nearly 2800 factors that were present in books that made the NY Times list but not present in the ones that didn’t. Yes, the NYT has issues with the ways it manages it’s list and it’s not a true sales meritocracy, but it is a powerful cultural signifier, and Archer, a former Penguin UK editor, wanted to better understand the differences.

The Bestseller Code is an exercise in finding meaning in those differences.

I don’t know if you remember when the book came out last year, but I do. I scoffed at it. Computer analysis? Of a creative endeavor? Please.

But that interview, flawed as it was, piqued my interest, so I borrowed a copy from the library.
It turned out to be interesting stuff.

What we talk about when we talk about luck

First, I want to say that the technology Archer and Jockers deployed—sentiment analysis, topic modeling, and more—was pretty impressive. The field is more advanced than I would have guessed.

Second, it turned out that the way they applied those tools, and the conclusions they drew from them, were entirely unremarkable. (Bolded because I want folks to take note.)

It’s common for folks to talk about success in the arts as part skill, part talent and part luck. I’ve talked at length on this blog about my opinion of “talent”, and at a little less length about “luck.” The effects of luck have been proven experimentally.

My question is always: what are they calling “luck”? What confluence of choices and incidents brought about this fortunate outcome? Because, to me, “luck” is what you call a series of events you don’t understand well enough to predict or control.

But what if we had the tools to look at things more closely? What if we had a better understanding of the differences between what people want to read and what we’re offering? What if we could narrow that gap?

Data doesn’t frighten me. Nihil veritas erubescit.

Besides, I’m a published writer with starred reviews and even, if you can believe it, fans. I already have the skills I need to break through to a larger readership, don’t I?

This is where my agent comes in

As I was reading The Bestseller Code, I kept thinking My agent could have written this.
Let me take an example. Using topic modeling, the algorithm breaks down what each book is “about.” Maybe a certain percentage might be concerned with crime and police work, and a smaller percentage for domestic matters. The next smallest percentage would concern, say, hospitals and medical concerns.

It seemed weird to me that algorithms are sophisticated enough to manage this task until I remembered Pat Rothfuss talking about programs that could handle the task five years ago.

Anyway, the books that sold well had fewer topics (around four), and those topics offered opportunities for dramatic contrast. Books that didn’t sell as well had more topics (around six, if I remember correctly). The subjects were more wide-ranging, less unified.

What’s more, one of the most important predictors of success was that a book devoted a certain amount of time to human interaction and connectedness. If one of the four topics was characters being with the people they cared about, living their lives and dealing with each other, that was a strong indicator of good sales.

Guys, my agent has been trying to teach me these lessons for years. For my whole career, I’ve been trying to establish relationships between characters the way a movie would: with a single, significant gesture or remark. She has been telling me, book after book, to give them more time on the page. To let them relate to each other. To let them bond. It turns out that human interaction in fiction is incredibly powerful, and I’ve been giving it short shrift.

She has also told me—many times—that I need to simplify. Often times I have too many storylines, plot turns, or characters. Especially characters. Too many “topics.” Maybe my work would reach a larger audience if it was more unified.

Another thing the algorithm does is generate plot curves through sentiment analysis. When the language of the book is full of upbeat words, like succeed, kindness, rest, and peace, the plot trends upwards. When it’s full of words like loss, failed, grief, and pain, it trends downward.

What surprised me is that, when the algorithm studied bestsellers, it produced plot curves very similar to the ones writers see all the time. One is quite similar to Freytag’s Pyramid; others matched different but fairly common models.

I’ll admit that I was startled to see a computer pull the old tried-and-true plot diagrams out of bestselling books, and how non-bestsellers seemed so flat. It made me question how well I manage the rise and fall of a plot curve and whether the language I use is appropriate for it.

There were other findings beyond those, obviously. The data was all descriptive, and it covered books that were popular but critically derided as well as popular but prize-winning. Except for a few surprises, like the need for scenes of human connection and a general distaste for sex (::shakes fist at America::) it’s standard stuff. Create a character who really wants something. Have them go after it. Make the plot turns powerful. Keep things focused. Write in a naturalistic style. Hook them in the first few pages.

Honestly, my agent could have written this advice, and as I was coming to the end of the book, it occurred to me that she sort of already had.

In which I step back from my edit notes to examine my edit notes

Just last week, my agent got back to me about a book I’d sent her. The news was bad, I don’t mind admitting, and of course she had some notes to give me.

As I was thinking about how closely the advice in Archer’s and Jocker’s book matched what my agent told me, I got the idea to go back through all her editorial notes for all of my books and look for patterns.

I’ve been happy to take her input—I signed with her, in part, because I knew she’d help make my work better—but I’ve been looking at them case by case. Book by book. It never occurred to me to look for trends.

To be fair, there was usually a year in between each new book, and sometimes more, and I’m a forgetful, disorganized person. It’s easy for me to carefully study a bunch of trees without once considering the forest.

So I opened all my old emails from my agent to review the notes she’d given me. My first thought was that past-me really needed to be more practical with his subject lines. My second thought was that I’d always thought of myself as a slam-bang thriller writer, a guy who could spin out an exciting story. It occurred to me that I wasn’t being exciting enough, because that self-conception wasn’t matched by outside reality. The work I was doing was earning fans and selling books (by my estimation, The Way into Chaos, which was self-published, has sold a little over 13k copies, which would be a fine, fine number for most NY genre publishers) but I wasn’t breaking through to the larger world.

What if I had placed myself in the “Good But Not Good Enough” category, and was missing out because I wasn’t really addressing persistent flaws in my creative choices?

So what were those persistent flaws? Obviously, each book had its unique problems, but there were several that popped up over and over.

Here they are:

  • Book started too slowly
  • Too many characters/plot complications/names
  • Characters not sympathetic enough/don’t have time for personal bonding

The hook must come sooner. More unity. More time for the characters’ relationships.

Honestly, I thought I’d already learned all the skills The Bestseller Code suggested I would need. I thought I was already working at that level. It’s pretty clear that I’m not.

The nice thing is that I’ll have a chance to be mindful of these persistent issues as I start a new book. Will it help? Shit, I hope so. I have ambitions, you guys, and I’m not meeting them.

My agent will still have notes for me, but maybe she won’t have to tell me the same old things she’s had to tell me every other time.

There’s more to say on the subject of computer analysis and the services various tech companies offer publishers, but that’ll have to be next time.

(If you thought this post was interesting or useful, why not share it?)

The State of the Novelist Address: I just sent a book to my agent

Standard

I thought I’d pop in and update things for folks, writing-wise.

First, earlier this week I sent a new novel to my agent. It’s a crime/mystery novel, a genre I’ve been reading for years. This isn’t my first attempt at this style, but it is the first one that I feel comfortable with. Some aspects of it fall right down the middle of the genre, while some are probably all wrong and will make me tear my hair out in revisions. We’ll see! But it feels good to start a book and send it to her in under six months. I’m not usually that prolific.

Which means I’ve returned to revision on my Twenty Palaces novella. I know I’ve been talking about this for a while, but this mini-book has resisted several attempts to write it. At this point, I feel I’ve solved most of the problems and hope to have it on sale before the end of the year.

Once I finish that, I’ll be working on something new. No idea what it will be, but I’m just going to pick an idea that sounds cool and run with it.

Thank you for reading this, and being here.

Randomness for 10/2

Standard

1.

2. How to control Alexa and Google Home through commands that are inaudible to the human ear.

3. A domino run in kaleidoscope: Beautiful. Video.

4. A Quick Beginner’s Guide to Drawing.

5.

6. Roald Dahl’s publisher threatened to drop him for being a jerk.

7. Why is it so hard to judge a screenplay from the movie that’s made from it?

Are you a writer hoping to improve their synopsis, pitch, and query skills?

Standard

Joshua Palmatier has put together a collection of essays on each subject. Remember link farms? Well, we have them again.

Here’s the one on writing a synopsis.

Here’s the one on putting together a short pitch.

Here’s the one on writing a successful query letter.

If you find those links helpful, please feel free to share.

Detective Twitter and the Case of the Unexpected Bestseller

Standard

I spent a fair portion of yesterday watching online amateur investigators look into an unexpected appearance on the NYTimes bestseller lists. Short version: a nerd-oriented site published it’s first YA fantasy, then identified bookstores that report sales to the New York Times and bulk ordered their own book.

It’s a time-honored tradition to try to scam your way onto the Times’s list, and for all the cultural cachet (not to mention the sales boost) that comes from putting “NY Times Bestselling Author” on the covers of your books, the process has plenty of flaws.

The Times itself curates it’s lists, leaving off the romance genre, for instance, because they would dominate any list that was truly fair. It’s a prestige thing, I guess. When the movie Julie and Julia hit the theaters, it bumped Julia Child’s Mastering the Art of French Cooking back onto the lists several decades after it came out. When the Lord of the Rings movies came out, the Times decided they weren’t going to list the books, no matter how well they sold.

The list is fudged in other ways, too: it’s compiled from the sales figures of a number of bookstores around the country, a list that’s supposed to be secret. It can’t accurately chart actual sales, because only the publishers have those figures, and they aren’t sharing.

Plus, book-buying is stronger at some times of the year than at others. You can make the list with lower sales in February than you’d need in July or December.

But still, it’s not a big deal to say that bestseller lists are imperfect. Everything created by human hands is imperfect, and imperfect systems can be exploited.

See also this article by an author who hired a company to buy enough copies of his business book to put it on the NYTimes list. The news articles about it have been vanished or are behind paywalls, but the author spoke candidly about what he did and why he did it.

One thing you notice is that the author wasn’t simply trying to sell more business books. For him, writing books was a stepping stone that would lead him to 5-digit speaking fees. Buying three thousand copies of his own book would be cheap compared to what he stood to make.

And reading through the detective work from yesterday, it concludes by saying that the author was expecting to turn the book into a film, and that she would be cast in the leading role. Once again, it’s not success in the book world these people are seeking. For them, books are a stepping stone.

So, sure, the lists are imperfect, but they still matter quite a bit. Not only are they worth a lot of publicity, they give negotiating power to authors when they negotiate with their publishers. But if you’re going to scam your way onto the list, be more careful than these people.

[Added later:] the author speaks to Huffpo, insisting that she didn’t game the system and that she worked to build buzz at Wizard World Comic Con events. She also claims there’s a bias against “new voices” even though her book bumped a debut novel by a black author that has been getting widespread buzz for months. So, yeah.

Going Big, Going Home, and Missing the Point: The Casual Hatred of Fun

Standard

Yesterday I tweeted this:

It’s not the first time I’ve tweeted that particular book cover, but it had been a while and it crossed my social media again, so I thought I’d make this point again: Don’t hold back. Have fun with your premise. People like fun.

A number of people have expressed doubt that this is a real book, but it is: The Angel Wore Fangs. If it sounds like a fun read, grab yourself a copy. Book seven of a series!

Now, you can glance at the stats on that tweet to see that it sort of blew up, 3700 RTs at the time I write this, and it’s only now slowing down. And my mentions have been flooded with quote tweets from people adding “Wow” or laughing smileys or whatever.

But some people have responded like fools.

First of all, if your first response to that back cover blurb is something along the lines of “And NY publishers won’t publish my books!” as though all mainstream publishing cares about is cheap trash when heartfelt human stories languish in rejection piles, I would suggest you’re learning the wrong lesson.

Simple fact: publishing is large and complex, putting out books for a variety of tastes. If an aspiring author is not writing light-hearted gonzo paranormal romance, the success of such a book has nothing to do with the lack of success Aspiring’s book has achieved. They’re in different markets, aiming for readers in a specific mood.

Instead of moaning, these Aspirings ought to be trying to learn something from it, like “Be fun.” And if “fun” is not your thing, then how about “people like fun.”

Even worse are the people who seem to think the author isn’t in on the joke. They call the description things like “train wreck” and talk as though the author is just piling random obsessions into a story without realizing it will make it funny. Guys, the author is in on the joke. Click the Amazon link above and read her bio. Assuming that she’s not making conscious choices about this is sexist bullshit. Unfortunately, it’s all too common, especially when the woman is writing paranormal romance.

My official stand on that blurb is that I think it’s amazing and hilarious and I’m a little envious. It makes me wish I were a romance fan because then I could write in the genre; the romance readership is HUGE. Instead, I’ve written nine novels, and not one of them has a decent romance in it. I’m stuck with the muse I’ve got.

What did make me happy, though, were the folks who took that blurb as inspiration. Reading that wild description seemed to give them permission to go a little wild with their own stories. At one point, someone tweeted that she wished she could have written that story herself, although it would have been “browner and queer-er.” I had to jump in to encourage her to do just that.

I spent much of last night and this morning skimming through my mentions, looking for people who seemed to need an encouraging word. I hope they go on to write their own.

And I’m sure that I helped Ms. Hill sell a few books. Hopefully, she’ll get a bunch of new readers out of it. (If you’re wondering, all those retweets have done nothing to sell my own books, but I wouldn’t expect them to.)

Anyway, I guess I should sum things up this way: “Fun! People like it.”

Discard Your Hair Shirts: Writers and Professional Jealousy

Standard

You’ve already seen this is you’re part of my Patreon, but here it is for everyone else.

It’s pretty common lately to see writers telling each other to stop being jealous of other writers’ achievements. “Don’t pay attention to them; pay attention to yourself.” is the common wisdom.

Now, I’m not going to argue that people shouldn’t focus on the things they can control; that’s solid advice. But just because professional jealousy can be expressed in toxic ways doesn’t mean it’s a good idea to tell people their thoughts and feelings are bad and wrong.

Professional jealousy is perfectly normal.

Everyone feels it from time to time. Everyone has to learn to manage that twinge when they hear about another writer finding a great agent, landing a publishing deal, making a best-seller list, whatever.

That’s the key. Not “Stop doing jealousy.” It’s “Jealousy is normal; how will you deal with it?”

Let’s break it down.

1. You feel jealous of a friend’s success

* Let’s say a friend has reached some milestone in their career that eludes you, and you’re burning with jealousy. What should you do?
* Do not tell your friend you are jealous. Your emotional responses are for you to deal with. It’s not your friend’s job to manage it.
* Acknowledge your jealousy. If their milestone is not a goal you are aiming for, try to shrug it off. If they’ve reached a milestone that you hope to reach someday, tell yourself you will work harder and smarter so you can maybe manage it, too.
* Then let your jealousy go. Experience it, but don’t hold on.
* Congratulate your friend. Something good has happened to them, and you should acknowledge that sincerely without making it about you.
* Talk to a neutral third party if you can not let your jealous go. Say the words “I’m happy for [friend] but it hurts that I can not manage to do the same.” With luck, your neutral third party will commiserate and you’ll feel better.
* It doesn’t have to be fair. Your friend’s success might be due to hard work and clever marketing on their part, but then again, maybe not. Life isn’t fair. But that’s not your friend’s fault, so don’t burden them with it. Just keep writing.

2. You feel jealous of a stranger’s success.

* Do not tell the stranger you are jealous. That’s weird.
* Acknowledge your jealousy. We all have milestones we want to achieve, and it hurts to fall short. That’s natural.
* Let it go. If you can’t, talk to a neutral third party. Say the same words as above.
* It doesn’t matter if it’s unfair. It doesn’t matter if you think the successful stranger’s work is trite, stuff, precious, derivative, or whatever. It doesn’t matter if you think they suck.

3. You can’t let go.

If you get to the point that you can’t interact politely with your friends and colleagues because of your jealousy, you should find someone qualified to help. It’s no different from any emotion that causes you to act inappropriately.

4. You can use jealousy constructively.

No, really. It’s possible, despite the way some people talk about it. We can use it to goad ourselves into working harder, or daring to try risky things. It can also spur us to venture into new areas, like self-marketing or online crit groups or who knows what.

But what we can’t do is use jealousy to squelch the perfectly natural urge to judge our success by the successes of those around us, or to see their success as a target we would like to reach someday.

So stop telling people not to be jealous.

It doesn’t work anyway, because humans have emotions and emotions can’t be reasoned with. It’s not even a bad emotion. It just sometimes spurs bad behavior.

Better to use your jealousy as motivation.

And yeah, I get jealous all the time. I just don’t make a big deal of it.

It’s how you spend your free time: the power of small decisions

Standard

One of my friends said something really smart on Sunday, and I thought I’d share it.

She and her partner live in Denver, and my son (who is 14) is planning to spend two weeks with them to pick their brains about Photoshop, After Effects, and a number of other programs they use. They make their living using all sorts of fancy software that I don’t know anything about, so he has a lot to learn in those two weeks.

ANYWAY. What she said, which I have to paraphrase because it was during an extended conversation, was: “What matters is how you spend your free time.”

To which I say: Yep.

Her story is that she was in college some years ago, learning software as part of her design class. I think it was Photoshop, but there was some cross-talk. Anyway, it was relatively new, and she and her friend were so fascinated by it that they spent their free time on a deep dive into the program, learning all the things it could do. In not time, the professor realized that she and her friend were more capable of teaching the software and asked them to do so. When she graduated, they offered her a teaching position.

It wasn’t because she was so good in class; it was because she was so engaged outside of it. The same is true of any kind of challenging field. If you want to be great in the arts, you have to cut out time from your daily life to practice and improve. That’s time you could be spending watching TV, going to the gym, sleeping in, playing video games, or making money.

If you click on the Tweet below, you’ll get a thread by comics writer Gail Simone on this very subject.

[Update: she deleted the whole thread. The gist was that people determined to be writers have to make the time to practice.]

I’ve tried to explain this to my son, because he acts like his great ambition is to be the best Overwatch player ever. It’s gotten to the point that I’m tempted to take away his computer games for good, even though he and I built a gaming computer for him just this past January. (Personally, I try to avoid most games because they’re addictive, and I’m vulnerable to that.) Choosing to spend all his free time playing video games is essentially choosing to be a regular joe with a joe job, and the US culture and economy squashes people like that now. If he’s going to be squashed, he ought to have the satisfaction of making art (or something!)

And what of myself? Thinking about spending down time always makes me audit myself, and I have to confess that I’ve been obsessing over Twitter and the election these past few months. It seems like my duty as a citizen to be as informed as possible, but how much of my time and energy do I REALLY need to devote to this? How much can I push off onto other citizens?

Clearly, I need to cut back and focus more on my work. The book I’m revising is complex and I need to get it to my agent so she can sell it. But Twitter is soooo tempting, almost like a video game.

And that’s the power of tiny decisions. Not the big stuff, like Where should I go to college or Should I quit my current job for that new one? No, the really important decisions are the huge clusters of tiny ones that we all make every day. Should I work on my book, or should I watch this tv show/go to the gym/hit the pub/etc?

Obviously no one can spend every spare moment of their lives writing (nor should they) but if you never choose writing over those other things that’s a clear statement of priorities.

[Added later: See also: Twelve Years from Hobbyist to Pro]