has joined #papers11-novel-mis-practice-based-methods

@joe.wright set the channel purpose: Paper Session 11: Novel MIs / Practice-Based Methods

has joined #papers11-novel-mis-practice-based-methods

has joined #papers11-novel-mis-practice-based-methods

has joined #papers11-novel-mis-practice-based-methods

has joined #papers11-novel-mis-practice-based-methods

has joined #papers11-novel-mis-practice-based-methods

has joined #papers11-novel-mis-practice-based-methods

has joined #papers11-novel-mis-practice-based-methods

has joined #papers11-novel-mis-practice-based-methods

has joined #papers11-novel-mis-practice-based-methods

has joined #papers11-novel-mis-practice-based-methods

has joined #papers11-novel-mis-practice-based-methods

has joined #papers11-novel-mis-practice-based-methods

has joined #papers11-novel-mis-practice-based-methods

has joined #papers11-novel-mis-practice-based-methods

has joined #papers11-novel-mis-practice-based-methods

has joined #papers11-novel-mis-practice-based-methods

has joined #papers11-novel-mis-practice-based-methods

has joined #papers11-novel-mis-practice-based-methods

has joined #papers11-novel-mis-practice-based-methods

*Crowd-driven Music: Interactive and Generative Approaches using Machine Vision and Manhattan* ? paper 49 *Design for auditory imagery: altering instruments to explore performer fluency* ? paper 20 *Curating Perspectives: Incorporating Virtual Reality into Laptop Orchestra Performance* ? paper 30 *Collaborative Mobile Instruments in a Shared AR Space: a Case of ARLooper* ? paper 37 *Reinventing the Noisebox: Designing Embedded Instruments for Active Musicians* ? paper 1

Paper session 11 will start in 15 minutes, start queueing up for the webinar!

Zoom: https://us04web.zoom.us/j/73767163818?pwd=bjNCU2tTNC9mVGMxdkR2ZGNzcEgxQT09

*external captions:* https://www.streamtext.net/player?event=NIME240720

live now! Come join us!

@channel don?t forget: when asking a question in response to a paper, please indicate in your message to which paper presentation you are responding. Either by mentioning the title of the paper or using the @ to direct it to the presenter. This will make it easier for people to follow the presentations and the Q&A later (due to being in different time zones). and please keep replies to the question in a thread!

@chris.nash on now

@chris.nash do the principles of agency, accessibility, aesthetic, and adaptive come from prior work, or did you articulate them from your own values?

the suitcase --> bass mapping is really clever :slightly_smiling_face: suitbass?

@chris.nash Good stuff! I appreciate that in the work shown you are working with a particular division of labour between composer and crowd, between sound-quality and control, and within particular genre commitments but have you experimented with the crowd having a more foundational spectro-morphological effect, e.g. where the organisation and behaviour of the crowd does synthesis?

Arriving at Bristol TempHop Meads !

Cool! @chris.nash You briefly mentioned something about the crowd starting to explore/interact with the mappings. Could you elaborate a little more on that?

Thanks for your presentation @chris.nash, this is super interesting. I wonder whether you collected any audience feedback to learn whether they appreciated their own agency in the system, or whether you observed any interesting behaviours as people become aware of the installation?

@a.guidi starting now

@n.s.robson I would have liked to gather feedback on the platform to support an empirical study, but the logistics of the setting wouldn't allow it - and the station were reluctant for us to be "canvassing", and those who stopped to chat still had somewhere to be. However, we got a lot of informal feedback that individuals definitely could hear their influence in a variety of settings - things only broke down when there were 100+ people, leading to five large equally spaced and sized groups where individuals (sharing their groups with ~20 others struggled to exert influence) - almost crowds of crowds. Fortunately, the overall musical effect was still pleasant enough, so the system degraded gracefully.

@lja Suitbass - love it!

@a.guidi More than a question a comment. I would argue that you would have different results with more experienced violinists. The short time span for the tasks would also have impact, as well as the lack of sccordatura tradition within violin playing. Changing tuning in Guitar for example is very common and a shift that is done without much thinking

@a.guidi There are some interesting ideas here; I wonder of the use of visual metaphors (i.e. ?imagery?) is most appropriate as opposed to explanation via other sensory modalities such as proprioception. It seems the latter is important in the first study you presented.

I wonder about the choice of notation as a means of specifying the auditory imagery to be realised - how could one do this test with professional musicians who play by ear, and what could results be?

do u think it is sometimes more beneficial to abandon imagery ...like to play by ear for some jazz musicians?

@lja is on now

@a.guidi also a comment, I'm a violinist and when I saw the non-transposed notation my head instantly hurt because when I see the notes I automatically thought of the fingerings that I would normally use. If I went note by note with a pencil I could figure out the transposition, but imagining trying to do it on the spot on the violin sounds hard; like trying to play in alto clef. Very cool study that you could map out these difficulties!

@a.guidi What do you think are the dominant (audio) features of auditory imagery for re-using the existing skills on a new interface?

@decampo Thanks for your question, Alberto. Beside the transposed notation and concert notation sections there was also an audio section. In the audio section, violinists tried to reproduce short music passages playing by ear on the retuned instrument. By analysing the data from this section, I did not achieve results that are statistically valid. However, I'm still in the data analysis process for part of this study, so there could be something to say for this section as well soon.

@a.guidi Very interesting research! I recently changed from Bb trumpet to C trumpet for the purpose of being able to play music for other instruments without transposing it and I am currently in the process of changing my mental imagery. Now that I have seen your presentation I have zero regrets!

Long winded follow up:

@imtortorken I do agree that sometimes is actually beneficial to abound imagery, a good example could be also certain types of electronic and contemporary music

@ko.chantelle thanks for your comment, Chantelle!

Any questions for @lja?

@lja hey Jack, nice presentation :slightly_smiling_face: Also like the piece, who is the author of the music modules? The entire group of performers?

@lja Great work ! I really appreciated the interconnection of the virtual and physcial spaces . How did the musicians feel about not being immersed in the virtual environment ? Would that be interesting for them in addition to the conductor ?

My comment was about the specificity of the results. Many professional string player I have collaborated with even without perfect pitch would struggle with the transposed notation as they would read the score and quickly audiate what the music should be. The example of the guitar was to emphasize that tradition is very important. In classical guitar it is very common to drop the 3rd string to F# for renaissance repertoire and many other changes for more contemporary music. In all cases, the written score is what the resulting sounds should be. At the same time, tablature is very common for guitar in non classical contexts, where all indication is the fret position and string. Changing tunning is trivial using tablature as the engagement is more in the motor actions to move fingers throughout the fingerboard with the music being a result rather than as clear a target.

@cagri.erdem that's a really interesting point, and for sure one of the topics I will focus in the next studies. For this paper, I mainly worked on pitch, but I would expand this research on articulation as weel as dynamic.

finally, regarding terminology: 'Audiation' is a commonly used term to signify what you called Audio imagery :slightly_smiling_face:

@lja thanks for highlighting alternative approaches for evaluating our work (e.g. qualitative, design-oriented, etc.)

agreed!

It really looks like an art game

Journey: https://youtu.be/61DZC-60x20 game mentioned by @niccolo.granieri

Reminds me also of Flower - really nice work @lja

Proteus (https://en.wikipedia.org/wiki/Proteus_(video_game)) is worth checking out, for people who are interested in games (broadly defined) as spaces for music-making

@sihwapark on now

Definitely! Looking forward for reading the paper and also for the follow-up studies.

and Flower and Journey came from the same studio! :slightly_smiling_face:

(this time in the _right_ place) I would have liked to gather feedback on the platform to support an empirical study, but the logistics of the setting wouldn't allow it - and the station were reluctant for us to be "canvassing", and those who stopped to chat still had somewhere to be. However, we got a lot of informal feedback that individuals definitely could hear their influence in a variety of settings - things only broke down when there were 100+ people, leading to five large equally spaced and sized groups where individuals (sharing their groups with ~20 others struggled to exert influence) - almost crowds of crowds. Fortunately, the overall musical effect was still pleasant enough, so the system degraded gracefully.

Proteus is fantastic!! Journey is great too. Ge?s main criticism of Journey is that its gamification of collecting scarf-lengthening icons takes away from your ability to practice peaceful _being_ since it adds a completionist _doing_ task overlaid on top of the experience.

@m.ortiz thanks for following up on this Miguel! I wouldn't say that transposed notation actually enabled violinists to perform with perfect fluency. So a certain amount of difficulty was still present. However, comparing the quality of their performance in the transposed notation condition, in the audio condition (reproducing short musical excerpts by ear) and with concert notation, transposed notation was the condition that lead them to better reatin their fluency. I do agree that tradition as well as the cultural space of the instrument is something essential in instrument design and skill repurposing. When choosing the instrument to perform the study, we chose the violin because it was more obvious / easy to measure the deterioration vs the retaining of performance qualities in the different conditions. Indeed, it would be interesting to replicate such a study with guitar, possibly looking at some other aspect of auditory imagery like articulation.

Keep the questions for @sihwapark coming!

I think there?s a lot of knowledge embedded in Journey for how to approach social interaction and cooperation with strangers, though.

There's an interesting balance between the simplicity providing the audience more liveness, versus more sophisticated mappings that support more ambitious musical constructs - respectively the agency of the audience vs. aesthetic of the composer. Certainly, for a public audience, the simpler the better, and there's lots of scope with part muting/unmuting (a kind of Ableton Live inspired progressive style of piece) but the whole piece can't necessarily all be simple mappings, because even a lay audience would perceive the result a ceiling or stasis in the piece.

Two questions: I may have missed it, but is it possible to share an AR space but be in different physical places? So a shared co-space? Second, have you considered whether the physical setting - inside / outside - might influence how people play with the tool?

this is super cool !

@sihwapark I would like to try it out, could you share the link here?

I answered this in the Zoom call, but I?ll repeat myself for the benefit of people reading later. In Stanford Laptop Orchestra (and probably many laptop orchestras), the rehearsal each week is used not only to practice, but also for the entire group to critique the work. So while I?m the primary author of the piece, the entire ensemble had a hand in its direction. One example is the design of the ?swirl and flick? gesture ? at first, I had the performers crouching and standing a lot, but they found this to be very uncomfortable! So I changed it in the final version.

Shared markers is what I was thinking?.!

if the AR virtually generated object could have ways to interact with real world objects that are also recognized by the app to make music, that may make the virtual object feel even more real?

like everyone with the app may poke the object...?

with their finger?

I answered this in the zoom call, but I?ll repeat myself here for people reading later. It was definitely a practical limitation that the performers were not in VR! The monitors they had available to them in the performance were smaller than what they had in rehearsals. But also, having everyone be in VR would enable an entirely different type of piece! It would enable people to interact with / against each other individually, rather than only working together collectively. Creating a VR laptop orchestra is mostly a question of funding at this point! Hopefully someone is working on it because it?s a really cool future. :slightly_smiling_face:

@john.sullivan2 on now

Keep the discussion going in the Slack channel!

@john.sullivan2 great start

Hey Jack, yes that would be amazing, but I think your shared virtual/physical space setup is also very interesting , and the immersion of all performers might lead to other issues (awareness for example)

@ko.chantelle Unfortunately, ARLooper is not published to App Store yet but it will be released later once it is stabilized. You could check later at https://sihwapark.com/ARLooper

Very true! Though that could maybe be mitigated by adding some conducting and directions into VR. The direction I was more interested in was actually putting audience members into VR. I wanted to have a few headsets that the audience could walk up to and look into briefly, taking turns, but it wasn?t really feasible with the large size of the venue. Now we?ll have to wait until the pandemic is over to try that!

Time to ask some questions to @john.sullivan2

Beautiful work @john.sullivan2! I may have missed it, but can you say some more about the types of performers and musical styles/genres you have been/will be working with?

:clap::clap::clap:@john.sullivan2

Also sharing VR headsets might be dangerous for a while :wink:

@john.sullivan2 This is a thoughtful approach. Did you differentiate between aspects best suited for the studio vs. the stage?

@john.sullivan2 Exploring the reasons why single maker's vs huge company's devices differ so much in outcome is essential - even if the different pathways of each are obvious.

@john.sullivan2 I like the piezo pickups!

(not a question)

@john.sullivan2 Are there demos of these we can see?

Go Miguel with some really excellent points!

what is a poster?

POSTERS AND DEMOS https://nime2020.bcu.ac.uk/posters-and-demos-24th/

shorter paper, more personal, interactive presentation

1000 points for super on time and clear host!!!

ah ha

With SLAM based markerless AR, it is difficult to synchronize two different physical environments. It would require the use of the same, shared markers. But if two spaces are identical, it would be possible to make a shared AR space. As an actual stage setup for AR based music performance, a wider, larger space could allow performers to more actively move around the AR space. But outdoor spaces could have limitations. For example, a sound speaker configuration would be harder than its indoor setup. And AR tracking requires a bright, stable lighting condition so outdoor might not guarantee it.

@imtortorken Thank you :slightly_smiling_face:

omg this has been happening the whole time?!

in parrallel?

@lja thanks for your talk. sorry about being slow asking a question - I had some IRL distractions! I think it's super interesting to think about how to engage with VR in LOrk settings. I wondered if you could say something about how the collective experience, which is often integral in LOrk performance, is considered when designing a VR work? My (limited) experience of VR is that it can feel quite isolating and foregrounds a personal experience over collective ones, so I wonder how to maintain the collectivity when you cant physically see your collaborators? e.g. In the piece you show I wonder how communication between conductor and performers is affected. And if everyone was in VR how would you have a sense of "togetherness" between performers, and maintain engagement with the audience, when you're not "present" in the same way?

It refers to the fact that at physical NIMEs there have been sessions where people stand with their posters:

Our main investigation has been around development of NIMEs for live performance, so as of now we haven?t deeply explored specific requirements for studio use. However we conducted an online musician survey prior to this work, which had interesting feedback about exactly this matter, contrasting different specifications for live and studio instruments. Definitely something to consider in the future.

Yes absolutely agreed, and there are many different aspects to this, from mass production/fabrication tools and techniques, to commercial marketing, etc., and of course the interconnected social/community aspects.

Haha thank you. Of course major credit and acknowledgement goes to @j.harrison @robert.h.jack @a.mcpherson and Strummi, which the pickup system was largely based off of!!

I?m also interested in this aspect of group music-making w VR: How can we facilitate that sense of togetherness, when different performers and the audiences are in different spaces?

It is a point where AR/MR musical instruments could stand out. Any physical objects could be (part of) AR musical instruments as controllers or compositional objects. Although ARLooper only allows unidirectional mapping between virtual content and physical environment (waveforms being fixed in locations), the physical objects can be used to interact with virtual content by scanning the objects before attaching virtual content.

I'm going to address this both as a researcher but someone that, as a musician, performs, studies and teaches in both strongly scored and strongly aural traditions. There is a huge context of training to this answer, in particular early training. People who are trained to focus on playing by ear (jazz, folk) will respond differently to people trained to play by score (classical). Training will make one or both of these methods really expedient. On top of this, it is also instrument dependent. A pianists knows if they are wrong before they've heard what they've played and in a lot of cases, a violinist doesn't know till they have heard it. The instrument designs require different visual/haptic knowledge. If you look at psychology studies of the effects of aural feedback on piano play for experienced classical musicians, the auditory feedback has actually surprisingly low relevance. Again, that is absolutely not true for a violinist (as shown in my own work). So yes, it might be true that it is beneficial for a jazz musician to abandon imagery, but a large part of that is due specifically to training.

Great presentation @john.sullivan2!

Realtime hand detection could be applied for this.

Lovely to see it in this format! Would be great to see/hear a demo

Unfortunately not quite at the moment, but on their way soon. The closure of labs and suspension of in-person participant research in March threw a bit of a wrench in the rollout of the instruments and the follow-up longitudinal study..

And yes, really great presentation! Very much 'up my street'

Oddly, my personal experience as someone who extensively learns music in both through visual and aural methods is how separate both approaches seem to remain in my brain. I learn a lot faster when there is a score, and I have no trouble memorizing a piece that I've learned by ear. But memorizing a piece I've learned visually is bizarrely hard, unless I go back treating it as an aural task (but still visualize the fingerings). I also played for years with classical musicians in an aurally taught ensemble (Balinese gamelan) and it is fascinating to see the different approaches the different people take/and even need to take, to learn the pieces.

Yeah @j.harrison, the instrument work and videos got a bit shuffled with lab closure and all, but plan to have them together soon and will share them with you!

Okay! Understandable. Sorry if you had already mentioned this. Is there a way we can continue to follow this? Website?

Yeah, it's like reading viola parts. At first it is so confusing! Then think, it's like the fingerings are just 3rd position and a few accidentals aside, it is no big deal. It's odd that that's the easiest way for me to sight read alto clef. (Okay, this is obviously a violinist perspective and I'm sure a proper violist would disagree.)

Thanks @a.mcpherson!

That?s a great question! It was definitely bizarre being a conductor that couldn?t see the ensemble. What we ended up doing was just a ton of practice and scaffolding around how the performance would proceed. During rehearsals I would both look at the ensembles with the headset off, as well as have the headset on and have another ensemble member off to the side critiquing how things were going. Then, during the performance, the way that I kept tabs on what was going on was equal parts (1) aural cues, hearing what?s coming out from spatially different parts of the speakers across the space, (2) visually seeing the virtual results of ensemble members? actions, and (3) trust! that the ensemble members would communicate and understand amongst each other with how to proceed! (For example, during the 1st movement the height of my body was a cue for the textural density of gestures, but the ensemble members had to coordinate between themselves to work that out.)

If everyone were in VR, then you would probably need to use a system of virtual avatars so that people could coordinate with each other. But that might further lock the audience out of an understanding of what?s going on, so it sounds like a really tough problem!

Thanks for the answer. I?ve worked in AR and we generally have to use markers with multiple spaces as you say. Since some of my work is for schools we?ve looked at ?toys? that the kids have to place before an experience starts.

And outdoors - yes, I get what you are saying, the lighting would be a problem except in the evening but of course in the evening seeing markers etc. is an issue.

Very interesting work, thx for presentation and for your answers!

(edit: wrong channel, pls ignore): This is super interesting! I was how this idea of push and pull effects can apply to programming environments and how people choose to build systems

ah ha - coded language !

@harri.renney Yes, at http://idmil.org/project/noisebox/. The page will stay updated with the project, including demo videos.

Hey @john.sullivan2, intriguing project! I noticed that the outcome of the workshops led to designs that, to my eyes, are notably different from the original Noisebox. Since this was a re-design approach?a concept that I love?I am wondering if you put any kind of limitations on the freedom of the participants, in terms of the extent to which the re-design ideas could have departed from the original instrument; and, more in general, what you consider an effective strategy to carry out re-design [shall we maintain the "essence" of the instrument at the cost of refusing a priori possible beneficial changes? Are there guidelines to "moderate" this process?]. You may have covered these points in the presentation and I will for sure read the paper, but I'd like to take advantage of the conference to hear your personal thoughts :slightly_smiling_face:

Hey @v.zappi, thanks! Yes definitely there was some thought put into this very question. I can?t really claim a theoretical basis for the re-design parameters, they were based on a couple practical things. For the workshops, the only real constraints were defined by the prototyping materials that the participants were given which, in general, tried to somewhat correspond with ?Noisebox-like? parameters, especially in size (e.g., sticking with something that was a discrete object that could be held in your hands) and materials (approximating available sensors, components and 3D print/laser cut parts). This was definitely done with an eye towards being able to replicate some of the workshopped ideas in the functional prototypes, without having to drastically reinvent the core foundations of the original instruments, like running on an embedded SBC, using easy-to-source components and fabricating the enclosures in-house.

On the other hand, a primary inspiration for the workshops was to break free of my own singular visions as a designer and harness inspiration from fresh ideas, no matter how wild or impractical they might be. In the end, the ?essence? of the original Noisebox may be mostly preserved in the underlying technical framework that the instruments are built on, as the function and sounds of each new version are quite different. Your question is, I think, an important one, and there is more to be considered and investigated in this area. The workshops themselves, and more of the theoretical research (beyond the 3 instruments), are being written up for a separate publication, and will definitely give this some more attention. Interested to hear your thoughts on this topic too!