Saturday, August 2, 2014

Money

I use lists of words to identify content in dreams. My original list for the category was tailored to matching the typical dream theme of finding money. It was a very low matching category. Most of my own money dreams were of finding money on the ground and picking it up with my hand. Other dreams were just about making change for a subway fare, awkwardly fumbling with coins in my pocket and difficulties counting the correct change.

I wanted to see how much people think about money how true to waking experience dreams of money were.

I started listing money-related words...

cost buy dollar sell debt expensive cheap luxurious tax rent bank account check book pay payment ATM credit card costly price valuable worth toll fee fare bribe reward reimburse wages free fiscal finance budget prize tarriff spend loan borrow earn wealth bargain investment wages coin dime nickel penny [0-9]+ bucks $ account salary rich poor savings cent retail sale dues cash retirement IRA IRS debit allowance bitcoin winnings

These all go into a regular expression which I test on some large sets of dreams. Context of the matches reveals a few problems.

"pay" mostly matches in context with 'attention'. Using a negative look-ahead removes most of these:
pay(?!( no| any| attention| much))

or only match pay (for|her|him|his|the)

bank is mostly used as in 'river bank', so I only match it with "account" and "book".

spend seems like a good money word, but it is mostly used like 'spend some time'.

My personal preference for the present tense means I forget to include bought, "sold" and -ed forms of money verbs.

$ is a non-word character and has a special meaning in regular expressions, so it has to be backslash escaped and preceded by a space in order to match.

\s\$([0-9\.,]+)?

And, of course, I forgot to include the word money in the initial list!

The final expression:

Money matches about 0.2% of words in most sets of dreams that I have tested. This is actually a fairly high occurrence. About the same frequency as the name of a close friend or family member. This underlies the importance of money in everyday experience.

Money is so pervasive it would seem to be a poor candidate for a dream-like theme. We might expect that it would be appear in dreams as a reflection of waking experience. It does, to a large extent, confirming the continuity hypothesis, but many appearances of money in dreams have a typically dream-like quality.

Three dream-like scenarios of money

Finding money: first a little, then more, gathering it in your hands till it becomes difficult to pick up any more.
Sometimes the found money is noticeably wrong, counterfeit, or of unknown nationality.
Found money often starts at low denominations, increases in value, changes to counterfeit.
Built in disappointment or rationalization with tacit realization of dreaming - you can't take it into waking life.
Trouble making change.
Anxiety about getting the right amount together when buying something.

Some example matches

From, now defunct, Sawlogs.net

724: "I have lived a long time and have had one dream three times over, a pleasant experience each time. I was a child of about 8 the first time I had the dream of walking through a forest and seeing pennies on the ground. I picked them up and continued to stroll. Soon I saw nickle, dimes and quarters, stooped down and scooped them up. I saw other people also finding money, picking it up in the same leisurely fashion, no rush, no feeling that someone else might get it all. It was a pleasant, happy dream, where I found a few dollars the first time, more the second time, and still more the third, which may have happened in my 50's. It is colorful, it is pretty, everyone is happy, no one is jealous; there is plenty for all."

2502: "As the dream began I found myself walking down a street that resembled that of Wall Street in New York City. My clothes were torn and shabby and I resembled a typical bum. I was asking the men (very wealthy looking) for a handout when they walked out the door of this one building which I was standing in front of. When finally one man came out who resembled my father and threw me a coin which I missed, it rolled down the sidewalk and into a little stairway that led to the basement of one of the buildings. When I walked down the steps where the coin had rolled I saw the entire stairway was filled with coins and bills of all denominations. I then leaped into the huge pile of money at the foot of the stairs and began to let the coins and bills run through my hands. I seemed to enjoy the feeling of having the money cover my entire body. I then began to stack the bills and coins very neatly in one corner when my dream ended."

From DreamJournal.net

100: This is the first time I was ever a shark in a dream, but that is a change from being a bat all the time. In this dream I was exploring a world. I was a shark, or a shark-like being. I still had hands, but we were all sharks in this place. There were shark malls, and restaurants, and the most interesting thing I remember about this world was the currency we had. There were two kinds of currencies, gold and red. Both were paper money, or something like paper, the gold currency being yellow, really. The red currency was printed on slightly smaller pieces of paper. I was in a restaurant and a shark couple sat with me. The male shark had just robbed a bank, but I'm not sure if the female had a part in it too, but I know the male did. Anyway I was pretending that nothing was out of the ordinary. The couple were ordering breakfast, and I was reading a paper, or pretending to, I was actually just curious about what was going on. I remember picking up a piece of the red currency, a $54 dollar bill. This was a large sum of money here, worth about as much as 50 dollars is to us in waking life. I know that the male shark dropped it accidently, but I knew he wouldn't miss it, so I kept it. The couple did not seem worried; they acted normal. I never saw any police sharks after them anyway.

510: This is part two to the dream "Bottle of Mice" I got to the bank. I was having trouble wth my clothes. They seemed not to fit me. It would seem that during the time I was walking to the bank my clothes changed all by themsleves. At one point I was wearing a skirt that narrowe too much at the bottom so I couldn't walk right, but that corrected itself after a while and I wore normal clothes. I got to the bank front counter and the lady behidn the counter asks if she can help me. I say I want to get some two dollar bills for this twenty. She says I need to fill out some forms first because it's illegal for anyone to carry the gold standard. I was puzzled, I said, "No, I don't want any gold, I want two dollar bills." She said two dollar bills were the gold standard. I decided not to argue with her, even though I had never heard of such rubbish in my life. She comes back with three two dollar bills. I know that this was not right, that I should get ten two dollar bills, but for some reason I was afraid to bring it up. So I accepted my three bills. The counter had a display where beneath it I could see some three dollar bills, which I thought was really weird. I had only seen them as a joke. I was supposed to fill in the reason why I wanted the two dollar bills. I was feeling embarrassed that I would have to explain that we were going to put them in our shoes for good luck. I hoped no one would actually read it.

Barb Sanders 1 set from DreamBank

1535: I am taking the wet clothes off a baby. She is lying on a bed and is a mess. I look for a clean shirt and find several. I choose one that's lacy and pretty, but hard to get on her. We struggle getting her hand in the arm and finally succeed. Then I take off her messy diaper and clean her up. She shits when I don't have a diaper on her. I clean that up, shuddering. Now my mother says, "Give the baby to me and you count all this money on my bed." I see piles and piles of coins (change). Most of them are foreign currency. What a tiring task. I say, "I'd rather hold the baby. I haven't had a chance to." She says, "Later, you can do it later." I sigh and start working on the money. Then I'm helping a woman pack. There are lots of snap shots laid out on the bed, some of Willie, some of her lover. I am concerned that her husband will come in and find out! I try to gather them up. Then I must hurry and I choose the shoes to pack, discarding several "Baby Jane" shoes for little girls.

Barb Sanders 2 set from DreamBank

409: ["Calling a friend for help."] I am trying to call Ernie to find out if his wife is at the University to help me Xerox my manuscripts. I can't find her. I am at a public pay phone. I am putting in coins, dimes, and they won't work. They keep coming back in the coin return slot. It is very frustrating. By now, I am sitting on the ledge the phone is on. I finally reach him. When I do, a large Mars bar pops into the coin return box as a reward. I get his answering machine. It goes for a long time and I patiently wait. Finally he comes on and I say About time!! We chat.

428: ["Scary dream conference."] I am at some university at a dream conference, seated at a table. I am working on my lap top computer. I call up some program and now can reach into the screen and pull out real money. Mostly pennies and some square coins, some Canadian coins, and a few dollar coins. As I do this, a man comes by, behind me and is showing some people some aura images on the wall behind me he can see. He says he has some evidence of the murders that happened here sometime ago. I feel a bit skeptical. Then I look and see the image of a woman on the wall and feel a bit frightened. Maybe that is the murderer's spirit. Ellie is somewhere in the background throughout the entire dream.

3268: It's time to downsize the Goodwill factory I'm in charge of. I see the crew of sewers threading their needles. They each choose a piece of used clothing to repair. They stand in a line before a big rectangular clothing bin, then they take them to the resale clothing store. I am going to the front door to open it. I have the key. As usual, several people are waiting to get in first. I open the door and hang up the key and notice I need the key to open the cash register. I have trouble getting the key string off the corner hook and then decide to wear it on my left wrist, even though it will be heavy and cumbersome. I get the bottom drawer open and I see fake 50 cent pennies stuck together. An older woman brings her purchases to me and I calculate it on the calculator because I can't get the cash register to work. It comes to $3.00 and some cents. She puts a pile of coins on the counter and I start counting. It's not enough because she's also put rectangular saint liberty (sweet liberty?) dollars there and they don't count as real money. One is a faded stained glass. A young woman is waiting, holding her crying baby! I say, "Let me hold her." I put her carefully on my shoulder, on a burp rag and pat her! It feels wonderful to hold a sweet baby. She quiets down. She's a newborn.

Kenneth set from dreambank

120: I am next to a building that I later recognize to be Mason, the Brimson college cafeteria. There is a kid there on rollerblades. I go over by a car. I find a ten-dollar bill. It is counterfeit. One side of the bill isn't printed. Then I find more bills, including 20's, 5's, and 1's, and I check the bottom sides to make sure they're real. The guys around me start scrambling to pick up the other bills. I shove a bunch in my pocket. One of the guys takes a $20 bill out of my pocket. I catch him and take it back. There are a few little kids trying to take photographs of me as I leave, as if I have done something wrong. I go into Carter School and a kid follows me in. I am careful not to let him get a photograph of my face. I go up the south stairs in the school. Teachers are there, some of whom I recognize. Finally I let the kid take my photograph, but I make him/her promise something before taking it.

499: I am in Mason Cafeteria at Brimson. My friend Brittany Johnson is sitting at a table trying to raise money for someone's grandma named Karla, who has died. I am picking up change on the floor (pennies, dimes,nickels, and quarters) and putting them in my pocket. A person I work with at the Brimson Library, name Don, is there. I say that it too bad about grandma Karla. I am upstairs in the cafeteria now.

644: We are on an island/mountain area. I go and explore. I find a crack in the side of a rock that leads to a cave. There is a lake or river right near the cave. The cave is on the edge of an island, like on Jap Lake. There is a white dock and a white airplane across from the island on the shore. The plane might be taking off. The dock seems to break in half, and form a "v". There are coins in and around the crack and cave. There are a lot of old pennies from the 1900's. I find a 1920D penny. There are others that are very old, and that I think are valuable.

820: There is a nicely cut area of green lawn with cement sidewalk and buildings, like a courtyard. There are pop machines there. A person on roller blades goes by. There is a skateboard in the corner by the pop machine. A boy comes to get the skateboard: it is his. I find three pennies, a $2 coin, and a $3 coin. A psychology professor I know from Brimson named Kari Cutting comes to me dressed in black and on crutches, to ask for directions.

Maya set from DreamBank

839: I'm not walking (or riding my bike?) along a road at nighttime. John is in front of me. I see him drop a huge wad of cash. I almost reach it when the wind snags it away from me in two different directions. I run after it, trying to catch it for John, angry that he let it drop. It is a lot of money, and it looks like more and more the longer I chase it along the ground. I see small bills, as well as large-amount bills ($50s). I finally catch up to some of the money and pick it up.

856: The women across the street (3rd St., in Ashland) are doing a yardsale. They have a simple display of a little table with a big sign that says "3 cents" (using the cent sign). I go look. There are little things for cheap. I see that the coins the women are using are special. The pennies, nickles and dimes are textured, with dents and scratches, since the women cannot see very well.

Alta set from DreamBank

988: and thank her, and ask for one of her cards - we only had her phone number, not an address, and don't know how far she had to come. Would be nice if she was local in case we need her again. She's very nice. I allow as how I need some stamps, so if we're not going to the party how about we go out and get some at the post office? We do that. Outside the building is a stamp dispenser that's a sit-down console where you put in your money and punch buttons. Rolly and I are talking away while I feed a dollar bill into the machine. It starts to work, and then the change starts to come out - but a whole lot of change comes out! I don't even realize it at first because I'm distracted talking to Rolly, then I see this shitload of change. Not small stuff, either, lots of quarters. Must be $30-$50 in here. I poke at it a bit, wondering how I'm going to pick it all up, and thinking maybe I should count it out by type, but a car pulls up and a couple of other people arrive - they're here for stamps too, and they'll be wanting to use the machine. O dear. How do I pick all this up without making a lot of noise and letting them see? They'll probably want to share - or maybe they'll expect me to turn it in - maybe I should - really I should, of course - but geez, you know, I really don't have to, no one would ever know. There's a piece of paper here, and while I'm dawdling (hoping they'll get impatient and go away)

Dorothea from DreamBank

314: I found in a volume of mother's 4 vol. Bible a wad of bills and two lots of stamps one of them purple 3 cents but I do not recognize the design. Believe it must be an old one. Wonder if it would be all right for me to use this money.

651: After we had climbed up to the green grassy level hilltop, I saw a shiny object. I called to him, "I've found a dollar." But then I bent it between my fingers and remembered father's thus bending counterfeit Mexican dollars in China. Then I hunted for a suitable spot to urinate. (and awoke)

Lawrence set from SDDb

65: I'm having tea and sandwiches in a cafe garden in Sri Lanka with Juan. It's my treat. This place feels more like my country than Sri Lanka in terms of the countryside. I ask for the bill. It comes to something ridiculous - far too expensive for Sri Lanka - about four times what it should be. I get very angry with the waiter and refuse to pay that much. He insists that is the price and ends up getting very upset, because he says his boss will take it out of his wages if we don't pay. So I give him all my money but I'm still slightly short, so I start scrabbling around on the ground under the table for dropped coins. I do find some, but one of them is fake - it's a very soft metal. So I finally leave owing the waiter the rest. I intend to return to pay him. Juan doesn't have anything to contribute.

Phil set from DreamBank

190: My wife Bonita and I were traveling in Russia, by train. We were at the train station, which was much like a subway station, fiddling with a Falk-type map to see where we wanted to go (a bridge across some river to some island in the river -- I couldn't find it on the map), and another Falk map to figure out which train to take to get there. It was really a mess. I thought I had time to study it a while, but while I was looking at it, a train pulled in. In the meantime, I had to exchange some money. I had American coins in my left pocket, Russian in my right, and I was afraid of mixing them together. I was standing at the exchange window while I was studying the maps, so I frantically began to try to figure out the exchange system, thinking it would be better, if possible, to exchange on a coin-for-coin basis. There were several types of Russian coins on display there, but some were in kopecks and some were something else, besides being in different amounts. There were also a lot of coins, including American ones, lying on the floor in front of the counter where people had evidently dropped them, and I couldn't resist wasting my precious time by picking some up. We finally decided we had better get on the train before it left, even if we weren't sure it was the right one. After we got on, I began trying to think of how to ask someone how to get where we wanted to go. But I couldn't think of the word for "station," nor the word and correct aspect for "go" in this case. Norman was suddenly there, and he said "station" was "some Russian word." I knew that meant "man," but thought it might also mean "station." I asked him whether it took the preposition and what he said didn't sound right to me.
48: I was going somewhere by taxi. It was to be a rather long trip, and I was already concerned about the cost. At some point, either on the way or after not finding our destination and heading back, either my daughter Christina or my son Burt, who was with me, spotted the Ferris wheel of an amusement park on a street at a higher level than we were on, and back the way we had come, and he/she wanted to stop. So I told the taxi driver to turn around, but he had to go a ways before he could do so, and then we didn't find any amusement park. I was anxious that the fare was going to cost even more. When we finally stopped, I was relieved that the fare was less than $5.00. The driver was Indian. For some reason, he gave me $5.00 and showed me his ID, and I was trying to figure out how to allow for the fact that he gave me $5 in trying to get the right amount out of my wallet to pay him and tip him generously. I finally decided just to give him back his $5.00 and give him $6 or $7 more-but my wallet was stuffed with big bills, and I couldn't find a $5. I decided to take everything out so that I could go through them more easily. In the process, a lot of checks fell out, and some of them had been shredded. So there was paper everywhere, and my checks were scattered, and it was a big mess. Sue Bailey and Frances Reed were there, and one of them commented to the other in my hearing that it was amazing that I was really very professional in other contexts. I felt really embarrassed to look so foolish.

200: I wanted to limber up my arm and practice throwing for distance. Two female friends went with me to a park to catch my throws, but we realized that we didn't have anything to throw. We discussed what would work in lieu of a baseball or softball. One of them suggested a grapefruit, and I said that it would be too soft and too large. But it reminded me of how perfect lemons and limes had felt to me in the past for throwing, and I was willing to entertain the possibility of that. In the end, though, I left the women in the park to go and get a couple of baseballs, either from my house or from a store. It ended up being a wild goose chase; I had to take a bus, and then wasn't sure where to get off. I was downtown, and it was pretty far between stops, so I would have to walk a lot if I got off at the wrong stop. I was sweaty, and the bus was crowded; I felt uncomfortable for the people close to me. There was a little girl standing next to me; she was friendly and talked to me a lot. When I finally got off the bus, I had walked into something like a video arcade at the bus stop when I realized I hadn't paid my fare. This was a Japanese-style system where you pay when you leave. I rushed back and found the bus hadn't left yet. When the last passenger had gotten off, I got on and dug a whole lot of change out of my pocket and started giving some of it to the bus driver. I had the impression that the fare was 60 cents, but I couldn't distinguish one coin from another very well (even though it was American money)

Tehuti set

2591: This dream is confusing. It seemed I was upstairs in my brother Eric's room and I was digging through a chest, looking for something. But I couldn't seem to remember what I was looking for. I kept finding different sci-fi books like the magazine Analog or Asimov's Science Fiction. I pulled them out and dropped them beside me because I thought maybe I'd like to read them later on. I was kind of afraid of digging further because I thought there might be bug crap and dead things and crawling bugs down there. That doesn't appeal to me very much. :P I kept finding silver coins all over the floor--all kinds of coins, but I only picked up the silver ones for some reason. Some had angels on them and I knew that they were like tokens; not meant to be spent, but I liked them. So I picked up only those coins. There were pennies and such too, but I wasn't interested in them.
4957: I had a bit of trouble with my money, as usual. I believe I started to seek out thirteen cents, but then reconsidered. What about tax? What if that was only a sale price and it in fact cost more now? Well, then I would have a little more money on hand. But it still could not cost an incredible lot. I think I decided on having like fifty cents or a dollar ready. I started looking for quarters; all I seemed to have was a handful of change. I remember perhaps a few dimes and nickels and maybe a penny or two, but I put those back and the quarters were what I focused on. I tried counting out four of them, I think; I planned on going up to the counter with several quarters, handing them over, and then receiving appropriate change. Better to have too much money than not enough ready! I might have changed my mind about how much I should have on hand, as I remember sorting through my change and picking out the quarters at least a few times over. I was being very compulsive about this; I wanted to be perfectly ready to pay.

5076: Then I thought over the last time something had been disconnected for nonpayment, how I might have saved the day if I had only summoned up the courage to go to the door and pay the bill myself. Maybe I could do that now? Did I have enough money? I suspected the overdue bill would be around $100. I think I had about that much. But I had to hurry. So I ran to my room and started rifling through the various bills I had lying on my little stand where I keep my money in real life, and started picking them all up. A wad here, a bill there, they were all scattered rather than in one big wad, but I kept scooping them all up, hoping I had enough.

I gathered what surely must have been around $100, but then saw more money sitting there, and started picking that up as well. And picking more up. And picking MORE up. I was surprised by now-- how much money DID I have here, after all? Surely more than $100! Maybe even as much as $300! I was impressed by my unexpected wealth--certainly I could pay that outstanding bill, and we would not be shut off. In real life I'm very phobic of answering the door or paying bills, but that fear was nowhere near as great as my annoyance at no cable would be in the dream, so I finally got all my scattered money in my hands and went running back out to the utility room...

Madeline from DreamBank

1460: I was in a parking lot. There was some sort of outdoor shopping spree. The first people to buy $200 worth of clothes would get them for $100. There was another guy (one of my online friends?) who I felt particular competition with. I went and picked out some clothes to try on, though I didn't know exactly why, since I didn't need new clothes or have extra money to spend. Then I was on a top floor of a building. The room was tan. I found two bills sitting in the corner- a $125 bill and a $150 bill. I was surprised and happy. Then a guy approached me- it was his money. I gave it back to him. I was sad to be left empty-handed. The guy said he was Peterson- I recognized him as my online friend Marilyn's love interest. I mentioned my disappointment and financial need, but he seemed not to care. I found another few pieces of paper on the floor and picked them up. Most of them were fake bills but there were two real ones in it, so I found about 6 dollars. I ended up in the elevator with him. Somehow he realized who I was, a friend of Marilyn's. He apologized for his curt behavior and told me his first name was Alan.

1500: My maternal Grandma Jane was driving, and I was in the passenger seat. She parallel parked in a bad neighborhood, in front of a bar or something. The parking brake was far over on the passenger side, and I had to pull it for her. I did so, but the car still rocked back and forth. We got out and in front of the joint was black plastic container with coins in it. I considered taking money out of it to buy a newspaper, and took a handful of coins. Then a person was leaving the place and I decided it was wrong. I turned and dropped all but one of the coins back in, counting as I dropped. 75 cents. I held on to one silver coin with permanent ink that said 3- 60 on it, indicating it's value of 60 cents. I decided it was okay to keep this one though as it was a foreign coin and unusable to the person anyhow. I dropped the coins in the bucket and the person leaving thanked me, thinking I was giving them money rather than simply putting it back. Then my mother got there. She asked me about the car parking and handed me the keys, asking me to move it. Where it was was apparently not acceptable and the 'correct' place to park it required a 3-point turn, which Grandma Jane couldn't do.

Money's typical dream-like characteristics

Money is ubiquitous in waking experience so it should be (and is) common in dreams. There's at least some evidence that mathematical thinking is impaired in dreaming. This may account for some of the characteristics of money dreams. An inability or difficulty to form a stable image of an object which incorporates both numbers and text may account for some cases where the dreamer reports dreamed money that changes from real to fake. Or it could be that we are taught to carefully scrutinize money for signs of forgery. When this degree of attention is applied to an unstable dream image metamorphosis results.

Tacit awareness of dreaming could also account for instances where the dreamer recognizes that the money is fake, counterfeit, or of unfamiliar origin. In a few examples the dreamer has to put the money back or give it to its rightful owner.

Also note that money is often found on the ground, on a floor on a bed. Flat surfaces with little context. One of my earliest hypnagogic visions was of a single Eisenhour dime, floating without context in a black void. As a seed to a fuller dream but not too fully developed. Runaway visions of money versus more life-like "math performance anxiety" caused by being in a brain/mind state where calcalations are difficult.

Friday, August 1, 2014

Flying

Flying is a seldom reported feature of dreams that is, nonetheless, considered a typical theme or feature.

flying: without mechanical assistance - unless the mechanical assistance is unrealistic: a flying car.

I looked at four sets of dreams where flying was fairly common to see if I could develop a regular expression that would match it with a high degree of reliability. I would like to match 80 % or more of references to flying without matching insects, planes, birds, etc. I'm really only interested in the dreamer's self reporting of flying, not on that of other dream characters.

Building the expression

The basics

fly|flying|flies (and maybe the grammatically incorrect "flys")
don't forget past tense: flew
(take|rook) flight
(float|glide|levitate) some of these might describe non-flying motion

I am only interested in dreamer flying

\b((we('re)?|I('m)?) (levitate?|soar|glide?|fl(ew|y|oat))(e?d|ing)?)\b

Misses "I was flying", "I began to fly", etc. So inset (\w+ ){0,4} = any string of word characters followed by a space. up to 4 of these, but maybe none.

Our generic word matcher (w+ ) should include a few more characters ([\w',-]+ ) to match hyphenated words, commas, contractions and possessives.

\b((we('re)?|I('m)?) ([\w',-]+ ){0,4}(levitate?|soar|glide?|fl(ew|y|oat))(e?d|ing)?)\b

I get 4 or 5 times as many matches. Now let's start looking at unwanted matches. (in a large set of dreams)

Then we see this little floating island/amusment
we flew the plane
As I walked over to the float,
and when they did i did a flying kick
The first bird I saw flew right in front of

Javascript's regex allows negative lookahead (?! ) but not negative lookbehind.

flying (?!birds) = match "flying" unless followed by "birds".

\b((we('re)?|I('m)?) (?!((was|were|am|are) watch(ed|ing)|see|saw) )([\w',-]+ ){0,4}(levitate?|soar|glide?|fl(ew|y|oat))(e?d|ing)?)\b

(?!\b(the|an?|(air)?planes?|jets?)\b)

(?!((was|were|am|are) watch(ed|ing)|see|saw) )

But this blocks "I saw that I was floating":

\b((we('re)?|I('m)?) (?!((was|were|am|are) watch(ed|ing)|see|saw) )([\w',-]+ (?!\b(the|an?|he|she|they|see|saw|(air)?planes?|jets?)\b)){0,4}(levitat(e?d?|ing)|soar(ing)?|glide?(ing)?|fl(ew|y|oat)(ing){0,1}))\b (?!([\w',-]+ ){0,3}((air)?planes?|jet|helicopter|bird|bee|insect)s?\b)

"I was in the air, flying"
So, add a second (?!) just before the flying verbs?

Final expression?

\b((we('re)?|I('m)?) (?!((was|were|am|are) watch(ed|ing)|see|saw) )([\w',-]+ (?!\b(an?|see|saw|(air)?planes?|jets?)\b)){0,4}(levitate?|soar|glide?|fl(ew|y|oat))(e?d|ing)?)\b (?!([\w',-]+ ){0,3}((air)?planes?|jet|helicopter|bird|bee|insect)s?\b)

Its only necessary to eliminate the most common false matches. Bees and helicopters might be overkill!

Also matches "I couldn't fly" But I want to know if the dreamer even expects to be able to fly. These could be eliminated by adding (\w+n't) to the lookahead.

Testing the expression

Go back to the expression for flying that doesn't specify who is flying.

Search for it in context.

I miss a few cases where the distance between "I" and "flying" is greater than 4 words.

In one dreamset the author describes other character's flying

I think he was floating
I think one of them flew
I believe they had to fly

So add (he|she|they|them) to the first negative lookahead.

\b((we('re)?|I('m)?) (?!((was|were|am|are) watch(ed|ing)|see|saw) )([-'\w,]+ (?!\b(an?|the|she|he|they|them|see|saw|(air)?planes?|jets?|birds?|bees?|insects?)\b)){0,4}(levitate?|soar|glide?|fl(ew|y|oat))(e?d|ing)?)\b (?!([\w',-]+ ){0,3}((air)?planes?|jet|helicopter|bird|bee|insect)s?\b)

Put the non-blocking words last in case the blocking words.

\b((we('re)?|I('m)?) (?!((was|were|am|are) watch(ed|ing)|see|saw) )((?!\b(an?|the|she|he|they|them|see|saw|(air)?planes?|jets?|birds?|bees?|insects?)\b) ?[\w',-]+ ){0,4}(levitate?|soar|glide?|fl(ew|y|oat))(e?d|ing)?)\b (?!([\w',-]+ ){0,3}((air)?planes?|jet|helicopter|bird|bee|insect)s?\b)

Al (skimmed from www.dreamjournal.net)

0.1113 % of words

Marco (collected from alt.dreams)

0.0806

Pegasus (http://www.dreambank.net/)

0.1049

Paul (private collection)

0.0638

I did a Compare search around the matches. The only unexpected features across the four sets was an increase in color and geographic description. The most frequent fliers were all male.

In all four of the gender-pairs sets of dreams I tested males had 2 to 3 times as many reports of flying.

*Compare does a search for 100 content categories in paragraphs where the target expression matches. It then does a log-likelihood comparison against the non-matching paragraphs of the dream set.

baseline female 0.0318, male 0.054

college 1999 female 0.0227, male 0.0492

harris youth female 0.034, male 0.113

blind female 0.0139, male 0.0265

Validation

I used Kelly Bulkeley's baseline set (http://kellybulkeley.org/creating-baseline-studying-patterns/) to search for the plain flying expression.

Perhaps 75 of the matches that I've marked with a + or ?

total: 171
. one in which im flying of course. others
, and I wanted to fly to New Hampshire
of dead women floating upon a river,
floor window and floated down the roof of
up with a paddle floating down this river
all. I was to fly, a fact which has +
vague. I was to fly alone ?
water out to a float. When I arrived
it seemed I was flying along in the air +
and couldn't fly anymore and I had
why Babs was flying it. The plane was
too. Kids went flying all over. I
In my dream I was flying, above the trees +
knowing I could fly when I wanted to +
garbage debris floating around on it. I
orange peelings floating around and other
I was flying in the sky, it +
how it feels to fly like a bird"". I
when I was flying and that it was a +
the golden hawks flying in the beautiful
realized I could fly and nothing could +
being able to fly a broomstick, ?
was the joy of flying and the cool wind
learned how to fly
the dream did we fly or go by boat. I +
. I would fly through the air +
I dreamed I was floating over a huge +
bed and started floating above myself. I +
everything was flying up and horsers
there. He is flying his military
window and start flying through the sky.
of the house. I fly away and that's +
that i could fly not like a bird i +
. In it I could fly but not very well +
. I would be flying low (more like +
floating) over my
. There were floating candles in the
at the loud jet soaring above me, a red
as being able to fly or hover. In
log cabins that floated on the water, it
that were also floating and thinking,
sure. We would fly together to ?
airplane. We were flying through some
.Run very fast , fly. i was then i ?
mane and tail, flying like my hair. The
on a machine and fly the 7 miles to
I had dreams of flying (just me, no +
the sensation of flying. The joy I felt +
i jumped i would float high up as if i +
was flying. also i held an
me go far as in float far. It was a
them and they fly it back into new
had dreams about flying/falling. One +
My mom and I were floating on a mattress in ?
wish that I could fly. I blasted off +
ship while I was flying and my mom granny
my partner and I flying our immediate
my area and I was flying. I felt like I +
who saw me flying. I wish that +
and planned to fly it himself
I was flying over the ocean. I +
and military jets flying in. It was night
him in heaven, floating on a cloud. I was
dream about flying and/or falling--I +
would be flying high in the air,
California. I floated into a home (not +
stairs and then floated all the way down
me and I started floating but the monster +
flying dream, as far +
old. Able to fly, but tethered, ?
, or limited to floating/flying just ?
and it made me float in slomo
. I pretended to float in the sandbox
I could fly. However, I +
, more like flying than falling. If +
enough, I could fly through the halls +
I was flying on a plane with a
I'm a the plane flying just above the
the ability to fly short distances, +
were two things floating in the murky
over the sky and floating around the houses
into the air and fly away by flapping +
notice that I was floating in air. I was +
suit and I was flying very slowly over +
colors). As I was flying over the tree +
luminous ball floating rapidly through
of a specter floating over my bed. I
I took flight and soared above the +
off jean shorts floating in the air
little bubble and float up off the floor
in which I could fly without the +
of swim flippers floating to the shore. My
flippers. He was floating on his back with
kind of randomly flying around the room,
and being able to float in a blue sky and +
male pheasants fly across my path
car and rnded up flying up out the ditch
rolled 3x I went flying through the air +
that I was flying around the Ifill +
told her I could fly her up. We went ?
were. They both floated away. When I went
I was in the air flying around at +
it up I was floating through space - +
steel and i would fly out of the cart ?
in was an eagle flying. When I hit the
and dragons flying above looking for
into me then flying in fighter jet in
in formation, flying in an F-18
into night time flying whether in an
the Blue Angels flying it was a
commercial jet flying low and made a
hill the power to fly disappeared and I +
have not had the flying kind for years - +
I am assigned to fly from the new USS
week I dreamed of flying, like an eagle, +
arms spread and soaring effortlessly. I +
often have flying dreams but this +
always able to fly behind the ?
dream, just the flying and avoiding +
pastime û flying. I freed my mind
and was able to fly quite far before +
, such dreams as flying over the +
into space and fly way, way out into +
being able to fly by leaning (like +
that I can fly, and should just +
that I could fly. I would just +
. I couldn't fly really high, +
often I'd try to fly during my waking +
in which I could fly, at will, by +
ground and then flying, usually around a ?
me that we could fly (without the +
column and rocks flying out of the
. I would start flying (floating upright +
I don't remember flying in any particular +
, but after flying several hundred
from heights or floated in the air to ?
Dream of flying through my school +
is easier. I was floating not flying and +
to dream I was flying a LOT, I still do +
the dream I can float along like taking +
will often end up flying away so high and ?
out. The bird was flying away to feed my
a dream that four floating monster heads (
of their car, and fly to the mall,
human partner. i fly to my residence. +
house, music floated out to me, and I
Cub, and it was flying right at us. We
Enemy planes were flying towards Cleveland
loaded airplanes fly out of the
excited and I fly around the boat +
the towel I could fly like a bird. +
wires. In flying up from the
was assigned to fly a new
a blimp. It would fly back and forth
I was in a rocket flying to some body out
. He started to fly low out over a
of dream is flying over heavily +
Force and I was flying jets in Korea. I
I was flying a small single-
another plane flying overhead and
water, but still floating and we got it to
. I seemed to be floating not walking +
because it was flying perfectly. I just
if it wanted to fly out and I could
were many hats floating on the surface. I
a large airplane fly over my house and
as it was flying along it fired
out an army flying jacket. I had
of airplanes flying around, shooting
which were flying around level with
we were both floating in the clouds +

Note that style of narration reveals two new ways dreamers describe flying.

Neither use "I".

Dream of flying through my school
Flying above the ocean, not in an aircraft.
im flying

We can also see the the inclusion of 'the' in the lookahead will excluding at least one case

I was in the air flying around

Next run the full expression on the same set which gives 66 matches;

total: 66
super suit and I was flying very slowly over
in Cleveland, and I wanted to fly to New Hampshire.
first it seemed I was flying along in the air,
a canyon below. I got scared and couldn't fly anymore and I had
and knowing I could fly when I wanted to,
I was Superman. I flew through the air
I was flying in the sky, it was
was smiling when I was flying and that it was a
so afraid then I realized I could fly and nothing could
longer afraid and I flew away and escaped.
in the dream did we fly or go by boat. I
a paintbrush. I would fly through the air
I dreamed I was floating over a huge crater
of the house. I fly away and that's
outside. In it I could fly but not very well
of my childhood. I would be flying low (more like
for sure. We would fly together to
in an airplane. We were flying through some
At about age 14 I had dreams of flying (just me, no plane
but everytime i jumped i would float high up as if i
. Then My mom and I were floating on a mattress in a
space ship while I was flying and my mom granny
my partner and I flying our immediate
I was flying over the ocean. I
was nothing that I could do so I flew out of my body and
and/or falling--I would be flying high in the air,
often. One night I dreamed that I flew up into the sky
California. I floated into a home (not
was after me and I started floating but the monster
sandbox on shore. I pretended to float in the sandbox and
As a child, I flew in my dreams and
high enough, I could fly through the halls
I'd notice that I was floating in air. I was
were vague. I was to fly alone
, colors). As I was flying over the tree tops
DREAM ENDED AS I WAS FLYING STRAIGHT UP
. Suddenly, I took flight and soared above the
years in which I could fly without the normal
side of the room. I was kind of randomly flying around the room,
. In the dream, I flew from Minneapolis (
hit us broadside!we rolled 3x I went flying through the air
gave me the pill. I flew out of my bedroom
When I was 6 I dreamed that I was flying around the Ifill
in the sky. After I flew around a few times
way to the top. I told her I could fly her up. We went
them like wings. I gained flight and flew to my city's
101 Dalmations). I flew past the grand
I picked it up I was floating through space - no
steel and i would fly out of the cart
dream, I realized I was dreaming, so I flew to get there
faster. As I flew over the
my training I am assigned to fly from the new USS
effortlessly. I often have flying dreams but this
a shotgun but I was always able to fly behind the
into flight. I couldn't fly really high,
to show me that we could fly (without the
in rural Kansas. I would start flying (floating upright
100-150 feet up). I don't remember flying in any particular
now it is easier. I was floating not flying and looking out at
. I used to dream I was flying a LOT, I still do
in the dream I can float along like taking
. In other ones I will often end up flying away so high and
a human partner. i fly to my residence.
quite excited and I fly around the boat
of some sort. I seemed to be floating not walking around
badly. The next, I believe we were both floating in the clouds

At least 3 of these matches refer to flying on a plane or space ship so we can claim at most 63 correct matches = 88 %. This figure is likely to be lower in other sets of dreams

in Cleveland, and I wanted to fly to New Hampshire.
space ship while I was flying and my mom granny
In the dream, I flew from Minneapolis

So add (?! (to|from)) to the final lookahead?

Conclusion

We constructed a probably over-complicated expression for matching flying in dreams. It finds 88% of basic references to flying while blocking most false matches to flying in planes, or observing flying objects.

We've also discovered that males are more likely to report flying dreams and that reports of flying dreams contain more color than expected.

The final expression: breakdown

\b((we('re)?|I('m)?) the dreamer

(?!((was|were|am|are) watch(ed|ing)|see|saw) ) not watching

((?!\b(an?|the|she|he|they|them|see|saw|(air)?planes?|jets?|birds?|bees?|insects?)\b) ?[\w',-]+ ){0,4} not someone else, not things that fly

(levitate?|soar|glide?|fl(y|oat|ew))(e?d|ing)?)\b flying verbs

(?!([\w',-]+ ){0,3}((air)?planes?|jet|helicopter|bird|bee|insect)s?\b) not things that fly

Saturday, July 5, 2014

Characteristic terms

I want to identify words in a set of dreams that are especially characteristic of the set.

This requires an external comparison so I combined the Kelly Bulkeley "baseline" set (http://sleepanddreamdatabase.org:5000/dream/) with large sampling of Sawlogs (http://www.sawlogs.net/) and dreamjournal (http://www.dreamjournal.net/). A little over 11,000 dreams and 650,000 words.

Found the percent of words for all non-stopwords that occurred more than twice. Saved as word-percent pairs. Only 8726 words.

To do the relative comparison, find the percent of words (filtered as above) in a target set of dreams.

Save any of these that are two times as frequent than in the baseline set.

Words are scored as taregt_%/baselien_%. If the baseline_% is 0, then set it to 0.0001.

This prevents division by zero, and discounts lower frequency words in the target set.

The sorted results are the 'characteristic terms' of the target.

The results are put in the word list on the right as 'sense cluster' links.

The result area displays the top 100 terms with their best neighbors.

Results

Lawrence

As previously blogged, I made the following guesses about Lawrence; prefers male friends, not married, but has a daughter with his partner, possibly woring in or interested in the arts, lives in England.

These were made based on an "Elements" search consisting of 90 or more word list categories, as well a secondary searches for names and family structure. The new

Characteristic terms search gives us most of the same information without the use of predetermined categories.

The terms, by themselves can suggest features. For example Lawrence has several male names at the top of the list and no female names. The third term "eldest" tells us little, by itself, but it refers to one of Lawrence's daughters.

To provide a little context for each term, I show the closest neighbors of each term as determined by word co-occurence.

Ralph <- MALE NAME

Terry <- MALE NAME

eldest <- FAMILY

Toby <- MALE NAME

client <- WORK

scruffy

Sal <- MALE NAME

Reece <- MALE NAME

Terry's <- MALE NAME

Karl <- MALE NAME

Juan <- MALE NAME

organisation <- BRITISH

Saul <- MALE NAME

marketing <- WORK

exhibition <- ART

industrial <- WORK

mattresses

home'

entourage

victorian

institution <- WORK | ART

artists <- ART

fragmented

Initially

hostel

persuade

Adrian <- MALE NAME

Lanka

Sri

dodgy <- BRITISH

premises

quilts

Lawrence

gardens

martial <- "ART"s

Reece's

ritual

Julian

preparations <- WORK

offices <- WORK

disused

Amsterdam

contains

attackers

1930s

humiliated

Len <- MALE NAME

technical

accommodation

customers <- WORK

arrogant

creative <- WORK | ART

venue

tutor

gig

London <- BRITISH

partner <- FAMILY

centre <- BRITISH

Claire

Tyler <- MALE NAME

constructed

drummer

Tim <- MALE NAME

Perhaps

perhaps

roots

involves

suburb

colleague <- WORK

film

tiles

keen

Ron <- MALE NAME

grounds

supermarket

decor

bump

university

containing

cheer

Neil

colleagues

guitar

art <- ART

artist <- ART

pools

Victorian <- WORK

factory <- WORK

based

fashioned

products

studio <- ART

cheap

design

architecture <- WORK

enormous

chatting

Germany

spacious

drums

stereo

project <- WORK?

activity

awkward

Apparently

cases

bass

apologize

medieval

Ralph singer playing bass grass lives pick friends Saul Terry plays friend terry Reece brother people dream getting drunk hairdresser lottery

Terry friend left friends Saul Ralph Terry's dad dream either brother comes night plays Tyler pick out Reece turn other look

eldest cousins daughter partner perhaps out

Toby another friend mine dream Toby's girlfriend shop bit

"eldest" is found with cousins, daughter and partner. A quick Fuzzy And search finds 3 occurences of "my partner and my eldest daughter".

Ralph may play in a band.

Possible profession-related words are also high in the list; client, marketing, industrial, offices , customer, colleague(s),

Also words related to art; art, artist, exhibition, studio, design, architecture

Expansion reveals....

We can guess about lawerence being British from; organisation , dodgy, flat, centre and London.

Guess: Lawrnece is involved in architecture. He enjoys musis in bars.

---------------------------------

Jasmine set from SDDb

Top names are female. Celine is Celine Dion.

Music; song, chord, jazz, octaves, instruments, sings, horn

Music editing; mixer, audio, receiver, amplifier, faders, CD, wireless, digital, MIDI, tape, record, speakers, RCA

Lots of casual speech and "anti-swears": blah, doo, dee, gee, wee, diddling, gosh, whacko, dinky, screwy

An artifact: of using with a "curly" quote instead of ' in contractions and possessives; isn, weren, didn, shouldn, wouldn, ll, t, s, don, wasn, d, m

jasmine <= top word is an artifact: each dream in the set starts with a line identifying the dream

Sandra <= sister

notion

Elizabeth

Celine

mixer

Dougherty

Peavey

audio

blah

Dion

Astrid

Terri

Larry <= Uncle

Bird

receiver

amplifier

Arlene

isn <= another artifact: dreams recorded with ` instead of '

back-up

Media

Gardens

Jill

Polly

diddling

hadn

Allison

goofy

Sheryl <= sister

Janice

doo

faders

Spain

Josie

Mermaid

weren

Grammy

conditioner

Cassandra

Sing

Sam <= brother

wouldn

Wee

librarian

Vision

editing

gee

fortunately

Tyme

geese

Bancroft

Arts

didn

Ariel

Wilcox

bummer

ding

wireless

shouldn

Coliseum

whacked

surround

bleah

Gee

shrinking

doesn

cane

dee

VIP

boardwalk

VCR

Chelsea

instruments

screwed

sprinklers

Dawson

accompaniment

Frederick

digital

Ursula

activate

MIDI

doot

wired

intercom

couldn

don

wasn

Fletcher

parrot

ignition

Moran

seed

speakers

theory

batteries

fragmented

input

dawdling

chord

Sycamore

whacko

beep

desktop

siren

Christine

boom

feedback

Joyce

loft

Airport

Rene

Susan

offices

Department

Crystal

Claus

Thomas

Dome

Triton

RCA

whacky

tandem

tech

coloring

candies

Guidance

Johnson

Muffin

LED

jazz

octaves

clip

goat

Sir

Marylou

Pizza

celebrate

exams

projector

Lutheran

adjusting

adjust

catalog

Happy

diddled

Springs

aren

gosh

echo

darned

fifteen

poem

awards

porcelain

Doo

adaptor

Factory

Gloria

babysitting

Rhode

absurd

fussing

alarms

claustrophobic

Palladium

dinky

jam

jacks

calendar

craziest

Styrofoam

Bend

plugged

screwy

thereafter

Stan

burner

magnet

Alright

technician

hoo

bugging

Sebastian

dong

sings

attach

obstetrician

Devil

vase

Loretta

zoo

horn

system

Last

tape

recorder

Mom

Spanish

Building

somebody

Performing

gearing

tippy

Sunday, May 4, 2014

New URL

I accidentally killed my Google drive, restored it, but now REve has a new URL;

https://googledrive.com/host/0B-wBCHNSdFnHdFp6aVltOVlwTkk/

Or shortened:

http://gdriv.es/reve

Saturday, March 22, 2014

Networks

Finds all matches for an expression. Usually looking for words or short phrases.
Counts all places where one match occurs within a short distance of another match.
Arranges the initial matches - up to 49 - in a circle.
Draws lines between matches which occurred close to each other. The thickness and brightness of the lines is proportional to the number of matches.
The halfway point between two matches shows the number of matches (A->B + B->A) if their count is > 4.
Useful for names and places.

I tested the the Jasmine set (combined Jasmine A and Jasmine B dreams from SDDb) there is a strong association between three names; Sandra, Sheryl and Sam. Running a Family search only reveals Sandra and Sheryl as the dreamer's sisters.

I ran a Network search on the results of a Names search combined with an expression which matches 'family identifiers'.

The most prominent lines connect Sandra, Sheryl and Sam. Mom and Dad are also strongly associated.
Slightly fainter links run between Mon and Dad, the two sisters and Sam. From this it's an easy guess that Sam is the dreamer's brother.

Also note the strong connections between Uncle and Larry and between Aunt and Christine, suggesting they are a married.

Richard Schweickert, in "Properties of the Organization of Memory for People:
Evidence from Dream Reports" (http://web.ics.purdue.edu/~schweick/Properties.doc) found associations between characters in dream reports that suggest dream accurately reflect waking experience of the dreamer's waking social life. He looked at three dream sets from DreamBank; Merri, Arlie and the Natural Scientist.

Schweickert measured all characters in the sets. I only identify names and then add 'family identifiers' to help determine the roles of some names.

Schweickert also used co-occurrence within a dream as the basis for association between two characters. I use co-occurrence of Names within a 55 character window.

Here's the result for Merri, showing the highest degree names; Dora, mother, Rudy and Father. There is only a faint line between 'sister' (near top of net) and 'Dora' (4 names to the right) because there is only one match were the two are mentioned together.

Sunday, March 16, 2014

Android support!

A recent update to Android 4.4.2 fixes a bug in the HTML5 file reader that allows REve to work on Android devices: at least on a Nexus 7. The file reader is the thing that allows you to select a text file and load it into REve. Previously, you had to paste the text into REve to do any searches.

Sunday, February 16, 2014

Before and after

A simple search of all words occurring immediately before and after a target expression - usually a single word.

The result is something like a concordance, but all (up to 220 of each) of the neighbors are show sorted by frequency. More common neighbors closer to the target.

Each of the neighbors is a hyperlink to a new Before and after search with the neighbor moved to the center.

Here's an example from the barb sanders dream set.

The target is "station".

Note the many senses of station that are displayed.

More generic words have more neighbors.

Tuesday, February 4, 2014

Keyboard

I saw a post about Patrick Wied's Keyboard Heatmap (http://www.patrick-wied.at/projects/heatmap-keyboard/) and decided to make my own.

Basically split a text field into individual character and count how many of each are present.

But for a keyboard 'a' and 'A' map to the same key. So do '#' and '3'.

What other keys could I map?

Shift = a capitalized letter followed by a non-capital letter. The Shift also gets bump if any of ~!@#$... are present.

No effort is made to distinguish between Left and Right Shift.

Caps Lock? I assume Caps Lock is on if I find a sequence beginning with 2 or more capitalized letters. But, because the end of a Cap Locks sequence matches the Shift key condition, I need to subtract the Caps Lock count from Shift's. No effort is made to distinguish between Left and Right Shift.

Enter = the sum of \r and \n in the split text. (Split the text using /^[\r\n]$/gm and take the length of the array.)

Tab = the sum of all \t. These don't show up in the split so I search for them with a regex in the original test.

I also thought about mapping the Back key to the number of times a list of commonly misspelled words are matched.

I use some css to display the keys in a QWERTY (only) layout.

To show the frequency of usage I first find the most pressed key and scale the background color of all keys to it. The most pressed key (space) will be the brightest red.

You can see the count for each key by placing your mouse over a key. The count shows as a tool-tip.

Since most keys a much lower in frequency that the top keys I used a log scale for the final result.

I tested this with several dream texts and, not surprisingly, they show the same ranking pattern Space, E, A, T, O, etc.

Things to test DE, FR, ES, PR texts. These contain characters which my code does not handle.

Here's the Keyboard for a dream log.

Here's one for a javascript file.
The big difference is in the use of parenthesis, braces, semicolons, equals and pipes '|'.

Wednesday, January 29, 2014

Parts of Speech 2

Problems with contextual rules

'building' can be a noun or verb.

Initially not found in the lexicon so I added it as ["NN","VBG"] meaning it's most often a noun but can be a verb with an "ing" ending.

The first part of the POS Tagger converts anything ending in "ing" to VBG.

Later rules convert many of the VBGs back to NNs.

Here's all of the rules that convert VBG to NN followed by examples where the rule should apply;

["VBG","NN","NEXTTAG","VBZ"] = Change VBG to NN if the next tag is VBZ (present tense tense verb).
The building is demolished.

["VBG","NN","PREVTAG","JJ"] = ...previous tag is adjective.
The large building near Main Street.

["VBG","NN","SURROUNDTAG","DT","IN"]
The building on Main Street.

["VBG","NN","NEXTWD","room"]
I'll wait for you in the meeting room.
I'll wait for you in the building room.

["VBG","NN","SURROUNDTAG","DT",","]
The building, the red chair and blue rug.

This rule only applies to the word "setting".
["VBG","NN","WDPREVTAG","DT","setting"]
The setting sun glowed red.

I made a test file with the examples above and then ran it through the tagger.

Words tagged as NN

./. The/DT building/NN ,/, the/DT
The/DT building/NN is/VBZ demolished/VBN
./. The/DT building/NN on/IN Main/NNP
in/IN the/DT building/NN room/NN ./.
the/DT building/NN room/NN ./. The/DT
The/DT large/JJ building/NN near/IN Main/NNP
the/DT red/JJ chair/NN and/CC blue/JJ
and/CC blue/JJ rug/NN ./. The/DT
./. The/DT setting/NN sun/NN glowed/VBD
The/DT setting/NN sun/NN glowed/VBD red/JJ
entered/VBD the/DT apartment/NN building/VBG ./.

Deadlock

In broader testing I found cases where "building" was still being wrongly tagged as a VBG:

"I entered the apartment building."
the/DT apartment/NN building/VBG ./. "/"

The error was that "apartment" was tagged as a NN instead of a JJ.

The rule ["NN","JJ","SURROUNDTAG","DT","NN"] should have converted "apartment" to a JJ, but the tag for "building" was still NN. Likewise, "building" would have been changed to "NN" if the previous word's tag was "JJ".

I don't see an obvious way around deadlocks like this.

Problem solved!

Removing a pre-tagging step from the jsPOS code fixed the problem.

// rule 8: convert a common noun to a present participle verb (i.e., a gerund)

if (tag.startsWith("NN") && words.endsWith("ing"))

tag = "VBG";

Now both "building" and other nouns ending in 'ing' are correctly tagged as NN. Even "apartment" (before "building") is tagged as "JJ".

I'll need to check if any of the other pre-tagging rules cause problems.

Saturday, January 25, 2014

Parts of Speech (POS)

Parts of Speech (POS)

I've added Percy Wegmann's (http://www.percywegmann.com/) implementation of the Brill POS tagger: jsPOS to REve.

I changed the parser to split the text into words and punctuation. I also split contractions like can't, we're, I'm, she'll, he'd
into
can n't, we 're, I 'm, she 'll, he 'd and then added lexical entries for these;

n't JJ (adjective)
're VB
'm VB
'd MD (modal)

's is mapped to [POS,VBZ] (possessive ending, is-verb).

Then I added a few steps to the post tagging, notably to change POS to VBZ.

I also added tests in the tagger to check that contextual rules only change a word's tag when the new tag is in the word's list of possible tags.

The original code only applied 8 transformational rules. I added a set of ~200 contextual rules which I borrowed from my python code, in turn borrowed from Brill.

Includes a few tags that never match?

Probably less than 85 % correct tagging.

The lexicon is from the WSJ corpus and includes many buisness and finance terms that are unlikely to be matched in dream descriptions. I removed several 100s of these.

Allgemeine
TuHulHulZote,
"non-interest-bearing"
"property-and-casualty"
"junk-bond-financed"
yff
"F.S.L.I.C"
"Bonds-b"
"DIAL-A-PIANO-LESSON"
"J.J.G.M."
"Asia\\",
"Junk-bond"
"Junk-bonds"
"junk-bond"

The lexicon did not include "I" and the method for tagging CD (cardinal numbers) was replaced.

The result uses the color text icons described in my previous post.

If you hover the mouse over a tag the description is shown in a tool tip.

Here's a list of the tags and their parts of speech;

CC: Coord Conjuncn
CD: Cardinal number
DT: Determiner
EX: Existential there
FW: Foreign Word
IN: Preposition or subordinating conjunction
JJ: Adjective
JJR: Adjective, comparative
JJS: Adjective, superlative
LS: List item marker
MD: Modal
NN: Noun, singular or mass
NNP: Proper noun, singular
NNPS: Proper noun, plural
NNS: Noun, plural
POS: Possessive ending
PDT: Predeterminer
PRP: Personal pronoun
PRP$: Possessive Personal pronoun
RB: Adverb
RBR: Adverb, comparative
RBS: Adverb, superlative
RP: Particle
SYM: Symbol
TO: to
UH: Interjection
VB: verb, base form
VBD: verb, past tense
VBG: verb, gerund or present participle
VBN: verb, past participle
VBP: Verb, non-3rd person singular present
VBZ: Verb, 3rd person singular present
WDT: Wh-determiner
WP: Wh-pronoun
WP$: Possessive-Wh
WRB: Wh-adverb
!: Excalmation
,: Comma
.: Sent-final punct
:: Mid-sent punct
$: Dollar sign
#: Pound sign
\: quote
(: Left paren
): Right paren

Licensed under the LGPLv3 license

http://www.opensource.org/licenses/lgpl-3.0.html